[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:09 dist-test-slave-dd32 krb5kdc[1871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:09 dist-test-slave-dd32 krb5kdc[1871](info): set up 2 sockets
Sep 24 11:47:09 dist-test-slave-dd32 krb5kdc[1871](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 11:47:10.466691 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:10 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:10.627017 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43935
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:10.951854 1884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:10.952135 1884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:10.952253 1884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:10.959326 1884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:10.959424 1884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:10.959467 1884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:10.959545 1884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:10.959630 1884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:10.969457 1884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:10.970715 1884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:10.971972 1884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:10.985114 1890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:10.985162 1892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:10.985162 1889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:10.985718 1884 server_base.cc:1034] running on GCE node
I20260924 11:47:10.986811 1884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:10.988278 1884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:10.989496 1884 hybrid_clock.cc:648] HybridClock initialized: now 1790250430989408 us; error 80 us; skew 500 ppm
I20260924 11:47:10.989878 1884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:10 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:10.992769 1884 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:10.994189 1884 webserver.cc:460] Webserver started at http://127.1.210.62:40985/ using document root <none> and password file <none>
I20260924 11:47:10.994840 1884 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:10.994982 1884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:10.995280 1884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:10.998191 1884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "82028a33fc9d41f5a5ff625e18c70209"
format_stamp: "Formatted at 2026-09-24 11:47:10 on dist-test-slave-dd32"
server_key: "fb0ea058b330ee43a8c36b0257e8378d"
server_key_iv: "d1513d5a8bed83304a233071f86d443c"
server_key_version: "encryptionkey@0"
I20260924 11:47:10.998914 1884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "82028a33fc9d41f5a5ff625e18c70209"
format_stamp: "Formatted at 2026-09-24 11:47:10 on dist-test-slave-dd32"
server_key: "fb0ea058b330ee43a8c36b0257e8378d"
server_key_iv: "d1513d5a8bed83304a233071f86d443c"
server_key_version: "encryptionkey@0"
I20260924 11:47:11.003842 1884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:47:11.007483 1899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:11.008716 1884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:11.008894 1884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "82028a33fc9d41f5a5ff625e18c70209"
format_stamp: "Formatted at 2026-09-24 11:47:10 on dist-test-slave-dd32"
server_key: "fb0ea058b330ee43a8c36b0257e8378d"
server_key_iv: "d1513d5a8bed83304a233071f86d443c"
server_key_version: "encryptionkey@0"
I20260924 11:47:11.009105 1884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:11.034101 1884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:11.038249 1884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:11.038524 1884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:11.057530 1884 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43935
I20260924 11:47:11.057615 1950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43935 every 8 connection(s)
I20260924 11:47:11.058776 1884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:11.062047 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 1884
I20260924 11:47:11.062361 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:11.062785 1951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:11.064781 1864 external_mini_cluster.cc:1428] Setting key d1248a72991ac46982e941287dc21da7
I20260924 11:47:11.075981 1951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209: Bootstrap starting.
I20260924 11:47:11.078860 1951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:11.079865 1951 log.cc:824] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:11.082700 1951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209: No bootstrap required, opened a new log
I20260924 11:47:11.089962 1951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82028a33fc9d41f5a5ff625e18c70209" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43935 } }
I20260924 11:47:11.090314 1951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:11.090420 1951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82028a33fc9d41f5a5ff625e18c70209, State: Initialized, Role: FOLLOWER
I20260924 11:47:11.091079 1951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82028a33fc9d41f5a5ff625e18c70209" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43935 } }
I20260924 11:47:11.091363 1951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:11.091497 1951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:11.091670 1951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 11:47:11 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:11.094560 1951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82028a33fc9d41f5a5ff625e18c70209" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43935 } }
I20260924 11:47:11.095016 1951 leader_election.cc:302] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82028a33fc9d41f5a5ff625e18c70209; no voters:
I20260924 11:47:11.095654 1951 leader_election.cc:288] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:11.095779 1956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:11.097494 1956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [term 1 LEADER]: Becoming Leader. State: Replica: 82028a33fc9d41f5a5ff625e18c70209, State: Running, Role: LEADER
I20260924 11:47:11.098064 1956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82028a33fc9d41f5a5ff625e18c70209" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43935 } }
I20260924 11:47:11.098418 1951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:11.100948 1958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82028a33fc9d41f5a5ff625e18c70209. Latest consensus state: current_term: 1 leader_uuid: "82028a33fc9d41f5a5ff625e18c70209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82028a33fc9d41f5a5ff625e18c70209" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43935 } } }
I20260924 11:47:11.101390 1958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:11.102027 1957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82028a33fc9d41f5a5ff625e18c70209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82028a33fc9d41f5a5ff625e18c70209" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43935 } } }
I20260924 11:47:11.102357 1957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:11.108520 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:11.069784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.1.210.62:43935)
0924 11:47:11.071777 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:11.071792 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:11.071841 (+ 49us) server_negotiation.cc:407] Connection header received
0924 11:47:11.081548 (+ 9707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:11.081567 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:11.081920 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:11.082280 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:11.085037 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:11.086585 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:11.088194 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:11.088858 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:11.089063 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:11.095473 (+ 6410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:11.095480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:11.095493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:11.095526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:11.099912 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:11.100978 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:11.100984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:11.100992 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:11.101220 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:11.101791 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:11.101796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:11.101802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:11.102574 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:11.106145 (+ 3571us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:11.106462 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 11:47:11.106971 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1758,"thread_start_us":1504,"threads_started":1}
I20260924 11:47:11.110077 1961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:11.115525 1961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:11.124845 1961 catalog_manager.cc:1347] Generated new cluster ID: 6eb32343eff4458c8a455fb2b108a92c
I20260924 11:47:11.124943 1961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:11.140442 1961 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:11.141762 1961 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:11.151726 1961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82028a33fc9d41f5a5ff625e18c70209: Generated new TSK 0
I20260924 11:47:11.152719 1961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:11.204239 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43935
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:11.535168 1978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:11.535485 1978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:11.535655 1978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:11.542829 1978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:11.542964 1978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:11.543150 1978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:11.553475 1978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:11.554721 1978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:11.556016 1978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:11.568785 1983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:11.568981 1984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:11.569432 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:11.570494 1978 server_base.cc:1034] running on GCE node
I20260924 11:47:11.571080 1978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:11.571949 1978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:11.573160 1978 hybrid_clock.cc:648] HybridClock initialized: now 1790250431573127 us; error 47 us; skew 500 ppm
I20260924 11:47:11.573467 1978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:11 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:11.576563 1978 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:11.577869 1978 webserver.cc:460] Webserver started at http://127.1.210.1:38051/ using document root <none> and password file <none>
I20260924 11:47:11.578485 1978 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:11.578578 1978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:11.578817 1978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:11.581652 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "c03546cbb07148f8a9e59d88468961e3"
format_stamp: "Formatted at 2026-09-24 11:47:11 on dist-test-slave-dd32"
server_key: "dd4ba4f99653f85348767a7a959f876c"
server_key_iv: "9e19aaaf289bd4b9c46abcb0140ff6ef"
server_key_version: "encryptionkey@0"
I20260924 11:47:11.582347 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "c03546cbb07148f8a9e59d88468961e3"
format_stamp: "Formatted at 2026-09-24 11:47:11 on dist-test-slave-dd32"
server_key: "dd4ba4f99653f85348767a7a959f876c"
server_key_iv: "9e19aaaf289bd4b9c46abcb0140ff6ef"
server_key_version: "encryptionkey@0"
I20260924 11:47:11.587144 1978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:47:11.590404 1993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:11.591646 1978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:47:11.591800 1978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "c03546cbb07148f8a9e59d88468961e3"
format_stamp: "Formatted at 2026-09-24 11:47:11 on dist-test-slave-dd32"
server_key: "dd4ba4f99653f85348767a7a959f876c"
server_key_iv: "9e19aaaf289bd4b9c46abcb0140ff6ef"
server_key_version: "encryptionkey@0"
I20260924 11:47:11.591985 1978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:11.607084 1978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:11.611146 1978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:11.611404 1978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:11.612241 1978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:11.613508 1978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:11.613579 1978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:11.613720 1978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:11.613780 1978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:11.647919 1978 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:39865
I20260924 11:47:11.647962 2105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:39865 every 8 connection(s)
I20260924 11:47:11.649241 1978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:11.651502 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 1978
I20260924 11:47:11.651741 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:11.652431 1864 external_mini_cluster.cc:1428] Setting key f7618ed3bc79d279625c5050bfb5ad46
Sep 24 11:47:11 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:11.672076 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:11.652373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:41569 (local address 127.1.210.62:43935)
0924 11:47:11.653476 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:11.653483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:11.653916 (+ 433us) server_negotiation.cc:407] Connection header received
0924 11:47:11.656610 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:11.656614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:11.656689 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:11.656779 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:11.659215 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:11.659888 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:11.661800 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:11.662570 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:11.662724 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:11.666523 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:11.666528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:11.666532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:11.666561 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:11.669003 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:11.669718 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:11.669728 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:11.669731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:11.669838 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:11.670291 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:11.670296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:11.670299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:11.670581 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:11.670754 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:11.671506 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 11:47:11.671713 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":853,"threads_started":1}
I20260924 11:47:11.673146 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:11.652874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43935 (local address 127.1.210.1:41569)
0924 11:47:11.653761 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:11.653800 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:11.656295 (+ 2495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:11.657473 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:11.657487 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:11.658265 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:11.659040 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:11.659065 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:11.660047 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:11.660053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:11.661655 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:11.661675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:11.662772 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:11.662777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:11.662967 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:11.663659 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:11.663692 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:11.666302 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:11.669160 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:11.669171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:11.669188 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:11.669582 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:11.669944 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:11.669950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:11.669954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:11.670174 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:11.670746 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:11.670754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:11.671143 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 11:47:11.671448 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 11:47:11.671981 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":727,"threads_started":1}
I20260924 11:47:11.675123 2106 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43935
I20260924 11:47:11.675462 2106 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:11.676604 2106 heartbeater.cc:511] Master 127.1.210.62:43935 requested a full tablet report, sending...
I20260924 11:47:11.678730 1916 ts_manager.cc:194] Registered new tserver with Master: c03546cbb07148f8a9e59d88468961e3 (127.1.210.1:39865)
I20260924 11:47:11.680135 1916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:41569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:11.694571 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43935
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:12.013693 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:12.013964 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:12.014055 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:12.021024 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:12.021190 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:12.021389 2112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:12.030558 2112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:12.031677 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:12.032905 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:12.046058 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:12.046128 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:12.046136 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:12.046437 2112 server_base.cc:1034] running on GCE node
I20260924 11:47:12.047210 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:12.048055 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:12.049276 2112 hybrid_clock.cc:648] HybridClock initialized: now 1790250432049241 us; error 54 us; skew 500 ppm
I20260924 11:47:12.049587 2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:12.052537 2112 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:12.053882 2112 webserver.cc:460] Webserver started at http://127.1.210.2:43699/ using document root <none> and password file <none>
I20260924 11:47:12.054515 2112 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:12.054656 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:12.054949 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:12.057760 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "89d3759a7d4941619d87f7e7f7b946c8"
format_stamp: "Formatted at 2026-09-24 11:47:12 on dist-test-slave-dd32"
server_key: "64cef5bfe7452c49323dace49a7401f8"
server_key_iv: "85ed631d3aee24bbff8b1b7ca13a9073"
server_key_version: "encryptionkey@0"
I20260924 11:47:12.058478 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "89d3759a7d4941619d87f7e7f7b946c8"
format_stamp: "Formatted at 2026-09-24 11:47:12 on dist-test-slave-dd32"
server_key: "64cef5bfe7452c49323dace49a7401f8"
server_key_iv: "85ed631d3aee24bbff8b1b7ca13a9073"
server_key_version: "encryptionkey@0"
I20260924 11:47:12.063347 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 11:47:12.067008 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:12.068375 2112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:12.068526 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "89d3759a7d4941619d87f7e7f7b946c8"
format_stamp: "Formatted at 2026-09-24 11:47:12 on dist-test-slave-dd32"
server_key: "64cef5bfe7452c49323dace49a7401f8"
server_key_iv: "85ed631d3aee24bbff8b1b7ca13a9073"
server_key_version: "encryptionkey@0"
I20260924 11:47:12.068723 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:12.095829 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:12.100471 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:12.100692 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:12.101603 2112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:12.102751 2112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:12.102818 2112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:12.102895 2112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:12.102937 2112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:12.145891 2112 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:34435
I20260924 11:47:12.145937 2239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:34435 every 8 connection(s)
I20260924 11:47:12.147122 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:12.150887 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2112
I20260924 11:47:12.151075 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:12.151587 1864 external_mini_cluster.cc:1428] Setting key 4ee4df95cd6f0663181786ceb05e2bd2
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:47:12.167323 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.149689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:54187 (local address 127.1.210.62:43935)
0924 11:47:12.149861 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:12.149865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:12.150915 (+ 1050us) server_negotiation.cc:407] Connection header received
0924 11:47:12.153263 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:12.153268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:12.153345 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:12.153461 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:12.154883 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.155565 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.157506 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.157918 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.158051 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.161502 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:12.161507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:12.161510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:12.161539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:12.164314 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.165004 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.165017 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.165019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.165196 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.165611 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.165616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.165619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.165929 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:12.166080 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:12.166755 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 11:47:12.166973 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:47:12.168334 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.150042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43935 (local address 127.1.210.2:54187)
0924 11:47:12.150756 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:12.150793 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:12.152958 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:12.153691 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:12.153699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:12.154184 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:12.154724 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:12.154740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.155714 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.155720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:12.157367 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:12.157378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.158144 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.158150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:12.158372 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.159049 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:12.159070 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:12.161293 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:12.164450 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:12.164458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:12.164474 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:12.164880 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:12.165308 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:12.165313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:12.165316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:12.165495 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:12.166058 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:12.166066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:12.166406 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 11:47:12.166658 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 11:47:12.167185 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":243,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
I20260924 11:47:12.169878 2240 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43935
I20260924 11:47:12.170192 2240 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:12.170877 2240 heartbeater.cc:511] Master 127.1.210.62:43935 requested a full tablet report, sending...
I20260924 11:47:12.172312 1916 ts_manager.cc:194] Registered new tserver with Master: 89d3759a7d4941619d87f7e7f7b946c8 (127.1.210.2:34435)
I20260924 11:47:12.173194 1916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:54187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:12.192083 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43935
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:12.542048 2245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:12.542343 2245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:12.542508 2245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:12.550684 2245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:12.550819 2245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:12.551023 2245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:12.563292 2245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:12.564657 2245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:12.566227 2245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:12.580186 2251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:12.580256 2250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:12.580150 2253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:12.582458 2245 server_base.cc:1034] running on GCE node
I20260924 11:47:12.583072 2245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:12.584015 2245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:12.585264 2245 hybrid_clock.cc:648] HybridClock initialized: now 1790250432585198 us; error 56 us; skew 500 ppm
I20260924 11:47:12.585616 2245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:12.589185 2245 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:12.590562 2245 webserver.cc:460] Webserver started at http://127.1.210.3:34651/ using document root <none> and password file <none>
I20260924 11:47:12.591187 2245 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:12.591279 2245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:12.591519 2245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:12.594305 2245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "698d462c11bf4a8a88ec00ac46edec19"
format_stamp: "Formatted at 2026-09-24 11:47:12 on dist-test-slave-dd32"
server_key: "de4825c3cf1d9ed52b2b923cf989e9b7"
server_key_iv: "d08e2ea5319dc1bfc238da77cb5285cf"
server_key_version: "encryptionkey@0"
I20260924 11:47:12.594986 2245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "698d462c11bf4a8a88ec00ac46edec19"
format_stamp: "Formatted at 2026-09-24 11:47:12 on dist-test-slave-dd32"
server_key: "de4825c3cf1d9ed52b2b923cf989e9b7"
server_key_iv: "d08e2ea5319dc1bfc238da77cb5285cf"
server_key_version: "encryptionkey@0"
I20260924 11:47:12.600709 2245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:47:12.603974 2260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:12.605257 2245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:12.605381 2245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "698d462c11bf4a8a88ec00ac46edec19"
format_stamp: "Formatted at 2026-09-24 11:47:12 on dist-test-slave-dd32"
server_key: "de4825c3cf1d9ed52b2b923cf989e9b7"
server_key_iv: "d08e2ea5319dc1bfc238da77cb5285cf"
server_key_version: "encryptionkey@0"
I20260924 11:47:12.605538 2245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:12.630959 2245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:12.635296 2245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:12.635529 2245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:12.636397 2245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:12.638123 2245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:12.638197 2245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:12.638275 2245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:12.638317 2245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:12.675235 2245 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:45585
I20260924 11:47:12.675323 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:45585 every 8 connection(s)
I20260924 11:47:12.676458 2245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:12.678608 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2245
I20260924 11:47:12.678793 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:12.679164 1864 external_mini_cluster.cc:1428] Setting key f4620fe9e537b4ff0101b816d3a3c39d
I20260924 11:47:12.683164 2106 heartbeater.cc:503] Master 127.1.210.62:43935 was elected leader, sending a full tablet report...
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:12.697460 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.679574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:38411 (local address 127.1.210.62:43935)
0924 11:47:12.680072 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:12.680078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:12.681102 (+ 1024us) server_negotiation.cc:407] Connection header received
0924 11:47:12.683870 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:12.683875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:12.683956 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:12.684058 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:12.685722 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.686540 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.688031 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.688460 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.688553 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.691749 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:12.691754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:12.691757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:12.691785 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:12.694405 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.695144 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.695148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.695151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.695310 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.695817 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.695823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.695826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.696088 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:12.696211 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:12.696958 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 11:47:12.697190 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260924 11:47:12.698434 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.680120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43935 (local address 127.1.210.3:38411)
0924 11:47:12.680921 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:12.680960 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:12.683546 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:12.684247 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:12.684255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:12.684963 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:12.685575 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:12.685593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.686663 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.686667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:12.687914 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:12.687925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.688631 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.688635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:12.688829 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.689530 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:12.689551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:12.691568 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:12.694547 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:12.694556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:12.694574 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:12.695006 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:12.695414 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:12.695417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:12.695420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:12.695657 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:12.696210 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:12.696218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:12.696583 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 11:47:12.696835 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 11:47:12.697299 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":290,"threads_started":1}
I20260924 11:47:12.699836 2373 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43935
I20260924 11:47:12.700094 2373 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:12.700781 2373 heartbeater.cc:511] Master 127.1.210.62:43935 requested a full tablet report, sending...
I20260924 11:47:12.702142 1916 ts_manager.cc:194] Registered new tserver with Master: 698d462c11bf4a8a88ec00ac46edec19 (127.1.210.3:45585)
I20260924 11:47:12.702881 1916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:38411
I20260924 11:47:12.707806 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:12.790251 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.774517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.1.210.1:39865)
0924 11:47:12.775415 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:12.775422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:12.775451 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:47:12.775510 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:12.775517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:12.775736 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:12.775868 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:12.777959 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.778970 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.780138 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.780707 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.780822 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.784615 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:12.784625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:12.784640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:12.784673 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:12.787290 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.787855 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.787861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.787863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.787976 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.788313 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.788316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.788318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.788610 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:12.788810 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:12.789228 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 11:47:12.789536 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":588,"threads_started":1}
W20260924 11:47:12.791548 2021 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41496
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:47:12 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:12.891851 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.878173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.1.210.1:39865)
0924 11:47:12.878423 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:12.878429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:12.878801 (+ 372us) server_negotiation.cc:407] Connection header received
0924 11:47:12.879000 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:12.879004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:12.879067 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:12.879146 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:12.881052 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.881747 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.883010 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.883415 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.883502 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.887213 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:12.887218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:12.887221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:12.887251 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:12.889817 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.890338 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.890342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.890344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.890454 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.890755 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.890758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.890760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.890981 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:12.891143 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:12.891376 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 11:47:12.891522 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:47:12.893234 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 1978
I20260924 11:47:12.905561 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:12.894170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.1.210.1:39865)
0924 11:47:12.894373 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:12.894378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:12.894596 (+ 218us) server_negotiation.cc:407] Connection header received
0924 11:47:12.894687 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:12.894691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:12.894740 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:12.894812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:12.896028 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.897245 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.898444 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:12.898940 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:12.899029 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:12.901113 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:12.901119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:12.901122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:12.901149 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:12.903529 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.904063 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.904066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.904069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.904188 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:12.904476 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:12.904479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:12.904481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:12.904688 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:12.904840 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:12.905129 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:47:12.905273 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:47:12.906297 2101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:12.993983 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 1978
I20260924 11:47:13.010452 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2112
Sep 24 11:47:13 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:13.035043 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:13.011530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.1.210.2:34435)
0924 11:47:13.012027 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:13.012035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:13.012059 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:47:13.012116 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:13.012123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:13.012328 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:13.012540 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:13.013822 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:13.014820 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:13.016035 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:13.016568 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:13.016717 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:13.029971 (+ 13254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:13.029981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:13.029995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:13.030025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:13.032556 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:13.033190 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:13.033216 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:13.033222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:13.033346 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:13.033679 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:13.033682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:13.033684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:13.034009 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:13.034230 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:13.034476 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 11:47:13.034670 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":268,"threads_started":1}
I20260924 11:47:13.036307 2235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:13.118855 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2112
I20260924 11:47:13.140193 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2245
Sep 24 11:47:13 dist-test-slave-dd32 krb5kdc[1871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:13.155663 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:13.141235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.1.210.3:45585)
0924 11:47:13.141743 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:13.141750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:13.141778 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:47:13.141834 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:13.141841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:13.142043 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:13.142162 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:13.143306 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:13.144347 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:13.145636 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:13.146176 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:13.146279 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:13.150155 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:13.150165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:13.150179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:13.150219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:13.152807 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:13.153547 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:13.153557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:13.153561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:13.153732 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:13.154098 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:13.154103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:13.154106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:13.154446 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:13.154781 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:13.155052 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 11:47:13.155310 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
I20260924 11:47:13.156944 2368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:13.237915 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2245
I20260924 11:47:13.253937 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 1884
I20260924 11:47:13.255061 1946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:13.354384 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 1884
2026-09-24T11:47:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3978 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:13 dist-test-slave-dd32 krb5kdc[2395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:13 dist-test-slave-dd32 krb5kdc[2395](info): set up 2 sockets
Sep 24 11:47:13 dist-test-slave-dd32 krb5kdc[2395](info): commencing operation
krb5kdc: starting...
W20260924 11:47:14.428946 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:14 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:14.598403 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43861
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:14.908010 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:14.908372 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:14.908561 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:14.915848 2408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:14.916020 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:14.916146 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:14.916254 2408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:14.916354 2408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:14.926486 2408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:14.928308 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:14.930459 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:14.943526 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:14.943491 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:14.943571 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:14.945358 2408 server_base.cc:1034] running on GCE node
I20260924 11:47:14.946175 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:14.947485 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:14.948722 2408 hybrid_clock.cc:648] HybridClock initialized: now 1790250434948674 us; error 59 us; skew 500 ppm
I20260924 11:47:14.949056 2408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:14 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:14.952117 2408 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:14.953526 2408 webserver.cc:460] Webserver started at http://127.1.210.62:37181/ using document root <none> and password file <none>
I20260924 11:47:14.954181 2408 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:14.954308 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:14.954622 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:14.957544 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917"
format_stamp: "Formatted at 2026-09-24 11:47:14 on dist-test-slave-dd32"
server_key: "e357fb850fd9047616bfcaf757568d93"
server_key_iv: "71b85ffa650f3cb431275033eba7bbfc"
server_key_version: "encryptionkey@0"
I20260924 11:47:14.958323 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917"
format_stamp: "Formatted at 2026-09-24 11:47:14 on dist-test-slave-dd32"
server_key: "e357fb850fd9047616bfcaf757568d93"
server_key_iv: "71b85ffa650f3cb431275033eba7bbfc"
server_key_version: "encryptionkey@0"
I20260924 11:47:14.963137 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:47:14.966336 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:14.967520 2408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:47:14.967680 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917"
format_stamp: "Formatted at 2026-09-24 11:47:14 on dist-test-slave-dd32"
server_key: "e357fb850fd9047616bfcaf757568d93"
server_key_iv: "71b85ffa650f3cb431275033eba7bbfc"
server_key_version: "encryptionkey@0"
I20260924 11:47:14.967880 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:14.989595 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:14.993942 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:14.994246 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:15.011965 2408 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43861
I20260924 11:47:15.011987 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43861 every 8 connection(s)
I20260924 11:47:15.013464 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:15.017041 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:15.023730 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2408
I20260924 11:47:15.023947 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:15.024709 1864 external_mini_cluster.cc:1428] Setting key c97dd1af25f32e5c3c95e0dd7d7ca7b9
I20260924 11:47:15.027808 2475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917: Bootstrap starting.
I20260924 11:47:15.031157 2475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:15.032289 2475 log.cc:824] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:15.035466 2475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917: No bootstrap required, opened a new log
Sep 24 11:47:15 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:15.043991 2475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43861 } }
I20260924 11:47:15.044368 2475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:15.044490 2475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdfcbfbbfe444e29ac87c5eb87f8917, State: Initialized, Role: FOLLOWER
I20260924 11:47:15.045186 2475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43861 } }
I20260924 11:47:15.045418 2475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:15.045586 2475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:15.045786 2475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:15.047616 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:15.026691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.1.210.62:43861)
0924 11:47:15.027446 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:15.027457 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:15.027498 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:47:15.029239 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:15.029260 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:15.029621 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:15.029904 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:15.031640 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:15.033244 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:15.035096 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:15.035806 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:15.035943 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:15.040393 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:15.040402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:15.040416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:15.040453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:15.043578 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:15.044213 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:15.044225 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:15.044231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:15.044449 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:15.044890 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:15.044895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:15.044897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:15.045513 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:15.045802 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:15.046261 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 11:47:15.046632 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":351,"threads_started":1}
I20260924 11:47:15.048257 2475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43861 } }
I20260924 11:47:15.048712 2475 leader_election.cc:302] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bdfcbfbbfe444e29ac87c5eb87f8917; no voters:
I20260924 11:47:15.049302 2475 leader_election.cc:288] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:15.049436 2480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:15.051389 2475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:15.055940 2480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [term 1 LEADER]: Becoming Leader. State: Replica: 5bdfcbfbbfe444e29ac87c5eb87f8917, State: Running, Role: LEADER
I20260924 11:47:15.056473 2480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43861 } }
I20260924 11:47:15.058940 2482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bdfcbfbbfe444e29ac87c5eb87f8917. Latest consensus state: current_term: 1 leader_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43861 } } }
I20260924 11:47:15.059389 2482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:15.059262 2481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfcbfbbfe444e29ac87c5eb87f8917" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43861 } } }
I20260924 11:47:15.059556 2481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917 [sys.catalog]: This master's current role is: LEADER
W20260924 11:47:15.068310 2494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:47:15.068418 2494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:47:15.068550 2495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:15.070609 2495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:15.078513 2495 catalog_manager.cc:1347] Generated new cluster ID: 33635181331645cf9c52713d2bdeef33
I20260924 11:47:15.078613 2495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:15.087745 2495 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:15.089493 2495 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:15.105517 2495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bdfcbfbbfe444e29ac87c5eb87f8917: Generated new TSK 0
I20260924 11:47:15.106557 2495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:15.160183 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43861
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:15.472441 2501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:15.472711 2501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:15.472805 2501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:15.479990 2501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:15.480103 2501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:15.480216 2501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:15.489825 2501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:15.490990 2501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:15.492202 2501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:15.504946 2509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:15.504908 2506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:15.504981 2507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:15.506129 2501 server_base.cc:1034] running on GCE node
I20260924 11:47:15.506675 2501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:15.507545 2501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:15.508770 2501 hybrid_clock.cc:648] HybridClock initialized: now 1790250435508735 us; error 51 us; skew 500 ppm
I20260924 11:47:15.509105 2501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:15 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250435, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:15.512068 2501 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:15.513389 2501 webserver.cc:460] Webserver started at http://127.1.210.1:43379/ using document root <none> and password file <none>
I20260924 11:47:15.514053 2501 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:15.514147 2501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:15.514389 2501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:15.517047 2501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "a3858ce0362e47b4b9728e8d4d28f65a"
format_stamp: "Formatted at 2026-09-24 11:47:15 on dist-test-slave-dd32"
server_key: "e10261c9784c65c79b4dbb158a3cf471"
server_key_iv: "caaa4f80e6d280fd24bd507ace409b19"
server_key_version: "encryptionkey@0"
I20260924 11:47:15.517827 2501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "a3858ce0362e47b4b9728e8d4d28f65a"
format_stamp: "Formatted at 2026-09-24 11:47:15 on dist-test-slave-dd32"
server_key: "e10261c9784c65c79b4dbb158a3cf471"
server_key_iv: "caaa4f80e6d280fd24bd507ace409b19"
server_key_version: "encryptionkey@0"
I20260924 11:47:15.522444 2501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:47:15.525667 2516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:15.526918 2501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:47:15.527077 2501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "a3858ce0362e47b4b9728e8d4d28f65a"
format_stamp: "Formatted at 2026-09-24 11:47:15 on dist-test-slave-dd32"
server_key: "e10261c9784c65c79b4dbb158a3cf471"
server_key_iv: "caaa4f80e6d280fd24bd507ace409b19"
server_key_version: "encryptionkey@0"
I20260924 11:47:15.527274 2501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:15.550879 2501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:15.554884 2501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:15.555162 2501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:15.556057 2501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:15.557363 2501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:15.557432 2501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:15.557583 2501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:15.557644 2501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:15.589857 2501 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:42599
I20260924 11:47:15.589931 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:42599 every 8 connection(s)
I20260924 11:47:15.591148 2501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:15.595238 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2501
I20260924 11:47:15.595445 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:15.596258 1864 external_mini_cluster.cc:1428] Setting key cb284be352664fedb167913fa016de5b
Sep 24 11:47:15 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250435, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:47:15.612766 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:15.593553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:37599 (local address 127.1.210.62:43861)
0924 11:47:15.593990 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:15.593994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:15.594930 (+ 936us) server_negotiation.cc:407] Connection header received
0924 11:47:15.597444 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:15.597450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:15.597520 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:15.597632 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:15.599502 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:15.600173 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:15.602674 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:15.603360 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:15.603508 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:15.607145 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:15.607153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:15.607161 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:15.607203 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:15.609943 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:15.610625 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:15.610632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:15.610640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:15.610752 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:15.611149 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:15.611154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:15.611157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:15.611404 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:15.611547 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:15.612190 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 11:47:15.612395 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":245,"threads_started":1}
I20260924 11:47:15.613765 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:15.594055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43861 (local address 127.1.210.1:37599)
0924 11:47:15.594739 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:15.594774 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:15.597147 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:15.597822 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:15.597829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:15.598522 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:15.599329 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:15.599357 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:15.600329 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:15.600336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:15.602541 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:15.602558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:15.603572 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:15.603576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:15.603761 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:15.604428 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:15.604455 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:15.606924 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:15.610077 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:15.610085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:15.610100 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:15.610503 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:15.610856 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:15.610862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:15.610866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:15.611034 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:15.611542 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:15.611549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:15.611852 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 11:47:15.612102 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 11:47:15.612597 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":275,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:15.615229 2629 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43861
I20260924 11:47:15.615513 2629 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:15.616185 2629 heartbeater.cc:511] Master 127.1.210.62:43861 requested a full tablet report, sending...
I20260924 11:47:15.618183 2440 ts_manager.cc:194] Registered new tserver with Master: a3858ce0362e47b4b9728e8d4d28f65a (127.1.210.1:42599)
I20260924 11:47:15.620033 2440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:37599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:15.636179 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43861
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:15.947886 2635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:15.948153 2635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:15.948247 2635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:15.955602 2635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:15.955698 2635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:15.955812 2635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:15.965204 2635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:15.966341 2635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:15.967532 2635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:15.980089 2640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:15.980060 2641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:15.980124 2643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:15.981205 2635 server_base.cc:1034] running on GCE node
I20260924 11:47:15.981683 2635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:15.982457 2635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:15.983688 2635 hybrid_clock.cc:648] HybridClock initialized: now 1790250435983634 us; error 45 us; skew 500 ppm
I20260924 11:47:15.983990 2635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:15 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250435, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:15.986965 2635 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:15.988356 2635 webserver.cc:460] Webserver started at http://127.1.210.2:45705/ using document root <none> and password file <none>
I20260924 11:47:15.988976 2635 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:15.989151 2635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:15.989463 2635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:15.992492 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "1371be9313dd447c8664887c913ab856"
format_stamp: "Formatted at 2026-09-24 11:47:15 on dist-test-slave-dd32"
server_key: "3173bfff11c41ce2dee6bc0f86c6fabe"
server_key_iv: "f875987baf9a9e2efe7f1adf18e012cf"
server_key_version: "encryptionkey@0"
I20260924 11:47:15.993554 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "1371be9313dd447c8664887c913ab856"
format_stamp: "Formatted at 2026-09-24 11:47:15 on dist-test-slave-dd32"
server_key: "3173bfff11c41ce2dee6bc0f86c6fabe"
server_key_iv: "f875987baf9a9e2efe7f1adf18e012cf"
server_key_version: "encryptionkey@0"
I20260924 11:47:15.998615 2635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:47:16.001883 2650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:16.003278 2635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:47:16.003460 2635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "1371be9313dd447c8664887c913ab856"
format_stamp: "Formatted at 2026-09-24 11:47:15 on dist-test-slave-dd32"
server_key: "3173bfff11c41ce2dee6bc0f86c6fabe"
server_key_iv: "f875987baf9a9e2efe7f1adf18e012cf"
server_key_version: "encryptionkey@0"
I20260924 11:47:16.003687 2635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:16.021200 2635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:16.025727 2635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:16.026037 2635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:16.026988 2635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:16.028311 2635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:16.028407 2635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:16.028607 2635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:16.028684 2635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:16.061846 2635 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:36371
I20260924 11:47:16.061883 2762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:36371 every 8 connection(s)
I20260924 11:47:16.063171 2635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:16.071102 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2635
I20260924 11:47:16.071341 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:16.071754 1864 external_mini_cluster.cc:1428] Setting key 1b5995d53bee36c8f4cc9625acecd094
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250435, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:16.083909 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.065747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:48815 (local address 127.1.210.62:43861)
0924 11:47:16.065921 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.065926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.067102 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 11:47:16.068992 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.068998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.069119 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.069245 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:16.070730 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.071685 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.073643 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.074059 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.074196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.077980 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:16.077985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:16.077988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:16.078020 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:16.080613 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.081636 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.081640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.081642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.081780 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.082270 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.082273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.082275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.082528 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:16.082638 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.083518 (+ 880us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.083682 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:47:16.085649 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.066254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43861 (local address 127.1.210.2:48815)
0924 11:47:16.066950 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.066987 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.068674 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.069488 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.069496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.069995 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:16.070560 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.070577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.071848 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.071855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.073491 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.073506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.074295 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.074302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.074538 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.075261 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:16.075284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:16.077694 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:16.080771 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:16.080784 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:16.080817 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:16.081522 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:16.081892 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:16.081900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:16.081906 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:16.082159 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:16.082656 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:16.082668 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:16.083136 (+ 468us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.083455 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.084046 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
I20260924 11:47:16.087216 2763 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43861
I20260924 11:47:16.087512 2763 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:16.088207 2763 heartbeater.cc:511] Master 127.1.210.62:43861 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:47:16.089701 2440 ts_manager.cc:194] Registered new tserver with Master: 1371be9313dd447c8664887c913ab856 (127.1.210.2:36371)
I20260924 11:47:16.090448 2440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:48815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:16.112157 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43861
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:16.421842 2768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:16.422104 2768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:16.422197 2768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:16.429055 2768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:16.429208 2768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:16.429343 2768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:16.438648 2768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:16.439777 2768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:16.441006 2768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:16.453893 2776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:16.453958 2774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:16.453918 2773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:16.454846 2768 server_base.cc:1034] running on GCE node
I20260924 11:47:16.455399 2768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:16.456262 2768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:16.457475 2768 hybrid_clock.cc:648] HybridClock initialized: now 1790250436457434 us; error 54 us; skew 500 ppm
I20260924 11:47:16.457769 2768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:16.460623 2768 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:16.462167 2768 webserver.cc:460] Webserver started at http://127.1.210.3:40087/ using document root <none> and password file <none>
I20260924 11:47:16.462771 2768 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:16.462867 2768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:16.463105 2768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:16.465951 2768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "92894b8a823344b5b27faa2a67e971af"
format_stamp: "Formatted at 2026-09-24 11:47:16 on dist-test-slave-dd32"
server_key: "ec835d4b6c890b2831c12f6cf6d35506"
server_key_iv: "c275a5707f7c29b25929ae8af777d842"
server_key_version: "encryptionkey@0"
I20260924 11:47:16.466670 2768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "92894b8a823344b5b27faa2a67e971af"
format_stamp: "Formatted at 2026-09-24 11:47:16 on dist-test-slave-dd32"
server_key: "ec835d4b6c890b2831c12f6cf6d35506"
server_key_iv: "c275a5707f7c29b25929ae8af777d842"
server_key_version: "encryptionkey@0"
I20260924 11:47:16.471551 2768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:47:16.474852 2783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:16.476084 2768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 11:47:16.476266 2768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "92894b8a823344b5b27faa2a67e971af"
format_stamp: "Formatted at 2026-09-24 11:47:16 on dist-test-slave-dd32"
server_key: "ec835d4b6c890b2831c12f6cf6d35506"
server_key_iv: "c275a5707f7c29b25929ae8af777d842"
server_key_version: "encryptionkey@0"
I20260924 11:47:16.476452 2768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:16.495971 2768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:16.499861 2768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:16.500159 2768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:16.501157 2768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:16.502471 2768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:16.502547 2768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:16.502733 2768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:16.502784 2768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:16.534339 2768 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:43671
I20260924 11:47:16.534436 2895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:43671 every 8 connection(s)
I20260924 11:47:16.535617 2768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:16.536589 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2768
I20260924 11:47:16.536795 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:16.537354 1864 external_mini_cluster.cc:1428] Setting key c6a9776146a321021beb0546dcf97f2c
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:16.555701 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.538371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:50301 (local address 127.1.210.62:43861)
0924 11:47:16.538550 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.538556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.539899 (+ 1343us) server_negotiation.cc:407] Connection header received
0924 11:47:16.541766 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.541771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.541852 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.541969 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:16.543390 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.544098 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.545654 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.546104 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.546238 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.549506 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:16.549512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:16.549517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:16.549558 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:16.552542 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.553243 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.553247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.553249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.553358 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.553712 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.553716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.553718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.554065 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:16.554193 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.554789 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.554996 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:47:16.556075 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.539037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43861 (local address 127.1.210.3:50301)
0924 11:47:16.539760 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.539802 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.541512 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.542191 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.542199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.542696 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:16.543215 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.543231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.544240 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.544259 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.545520 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.545531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.546331 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.546340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.546569 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.547217 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:16.547237 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:16.549325 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:16.552677 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:16.552683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:16.552694 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:16.553134 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:16.553457 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:16.553460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:16.553462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:16.553617 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:16.554165 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:16.554169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:16.554453 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.554660 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.555058 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":291,"threads_started":1}
I20260924 11:47:16.557354 2896 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43861
I20260924 11:47:16.557639 2896 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:16.558297 2896 heartbeater.cc:511] Master 127.1.210.62:43861 requested a full tablet report, sending...
I20260924 11:47:16.559628 2440 ts_manager.cc:194] Registered new tserver with Master: 92894b8a823344b5b27faa2a67e971af (127.1.210.3:43671)
I20260924 11:47:16.560393 2440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:50301
I20260924 11:47:16.564400 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:47:16.583009 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.570079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.1.210.62:43861)
0924 11:47:16.570220 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.570224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.570714 (+ 490us) server_negotiation.cc:407] Connection header received
0924 11:47:16.570901 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.570940 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.571016 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.571113 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:16.572525 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.573276 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.574557 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.575126 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.575263 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.577894 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:16.577905 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:16.577908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:16.577935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:16.580530 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.581171 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.581177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.581182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.581310 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.581666 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.581674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.581677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.581949 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:16.582175 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.582505 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.582696 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:47:16.588537 2440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:16.595235 2440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:16.618451 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.606204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:42599 (local address 127.0.0.1:53398)
0924 11:47:16.607119 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.607168 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.607362 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.608687 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.608698 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.608739 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:16.609262 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.609279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.611970 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.611979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.615106 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.615126 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.617530 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.617536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.617752 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.617806 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.617902 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.618061 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":545,"threads_started":1}
I20260924 11:47:16.620414 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.606505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.1.210.1:42599)
0924 11:47:16.607952 (+ 1447us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.607964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.607997 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:47:16.608105 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.608115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.608409 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.609497 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:16.609758 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.611778 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.615378 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.617275 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.619070 (+ 1795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.619319 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.619387 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.619549 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":1163,"threads_started":1}
I20260924 11:47:16.622660 2629 heartbeater.cc:503] Master 127.1.210.62:43861 was elected leader, sending a full tablet report...
I20260924 11:47:16.624295 2564 tablet_service.cc:1467] Processing CreateTablet for tablet b1c5fc320e3f4d238b57a3b45d4e7a74 (DEFAULT_TABLE table=test-table [id=a63e1758bc13441a918383ac5d755171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:16.625825 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c5fc320e3f4d238b57a3b45d4e7a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:16.626864 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.607652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:36371 (local address 127.0.0.1:54654)
0924 11:47:16.618692 (+ 11040us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.618710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.618809 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.620184 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.620190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.620214 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:16.620467 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.620477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.622327 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.622332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.624185 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.624197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.625974 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.625980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.626228 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.626257 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.626309 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.626451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10922,"mutex_wait_us":36,"thread_start_us":12573,"threads_started":1}
I20260924 11:47:16.626912 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.606972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.1.210.3:43671)
0924 11:47:16.617458 (+ 10486us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.617470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.617901 (+ 431us) server_negotiation.cc:407] Connection header received
0924 11:47:16.617995 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.618002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.618229 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.619021 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:16.619285 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.621926 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.624338 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.625780 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.625923 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.626148 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.626433 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.626592 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10292,"thread_start_us":10112,"threads_started":1}
I20260924 11:47:16.626912 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.606399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43671 (local address 127.0.0.1:40420)
0924 11:47:16.617645 (+ 11246us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.617665 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.617789 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.618435 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.618441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.618470 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:16.618839 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.618852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.622059 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.622063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.624164 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.624177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.625996 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.626002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.626241 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.626285 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.626375 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.626502 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11128,"mutex_wait_us":69,"thread_start_us":11093,"threads_started":1}
I20260924 11:47:16.627982 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.608165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.1.210.2:36371)
0924 11:47:16.619215 (+ 11050us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.619223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.619259 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:47:16.619363 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.619374 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.619628 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.619855 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:16.620741 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.622186 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.624393 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.625801 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.627450 (+ 1649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.627607 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.627663 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.627773 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10900,"thread_start_us":10990,"threads_started":1}
I20260924 11:47:16.630642 2698 tablet_service.cc:1467] Processing CreateTablet for tablet b1c5fc320e3f4d238b57a3b45d4e7a74 (DEFAULT_TABLE table=test-table [id=a63e1758bc13441a918383ac5d755171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:16.630364 2831 tablet_service.cc:1467] Processing CreateTablet for tablet b1c5fc320e3f4d238b57a3b45d4e7a74 (DEFAULT_TABLE table=test-table [id=a63e1758bc13441a918383ac5d755171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:16.631933 2698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c5fc320e3f4d238b57a3b45d4e7a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:16.631968 2831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c5fc320e3f4d238b57a3b45d4e7a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:16.642841 2914 tablet_bootstrap.cc:492] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: Bootstrap starting.
I20260924 11:47:16.644722 2915 tablet_bootstrap.cc:492] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: Bootstrap starting.
I20260924 11:47:16.646188 2916 tablet_bootstrap.cc:492] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: Bootstrap starting.
I20260924 11:47:16.647317 2914 tablet_bootstrap.cc:654] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:16.649003 2914 log.cc:824] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:16.649080 2915 tablet_bootstrap.cc:654] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:16.649503 2916 tablet_bootstrap.cc:654] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:16.650111 2915 log.cc:824] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:16.650730 2916 log.cc:824] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:16.652136 2914 tablet_bootstrap.cc:492] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: No bootstrap required, opened a new log
I20260924 11:47:16.652849 2914 ts_tablet_manager.cc:1397] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 11:47:16.653354 2915 tablet_bootstrap.cc:492] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: No bootstrap required, opened a new log
I20260924 11:47:16.653621 2915 ts_tablet_manager.cc:1397] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 11:47:16.663110 2916 tablet_bootstrap.cc:492] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: No bootstrap required, opened a new log
I20260924 11:47:16.663372 2916 ts_tablet_manager.cc:1397] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260924 11:47:16.663928 2914 raft_consensus.cc:348] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.664342 2914 raft_consensus.cc:374] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:16.664465 2914 raft_consensus.cc:729] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92894b8a823344b5b27faa2a67e971af, State: Initialized, Role: FOLLOWER
I20260924 11:47:16.664865 2915 raft_consensus.cc:348] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.665112 2914 consensus_queue.cc:260] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.665287 2915 raft_consensus.cc:374] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:16.665380 2915 raft_consensus.cc:729] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3858ce0362e47b4b9728e8d4d28f65a, State: Initialized, Role: FOLLOWER
I20260924 11:47:16.666098 2915 consensus_queue.cc:260] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.667057 2914 ts_tablet_manager.cc:1428] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: Time spent starting tablet: real 0.014s user 0.009s sys 0.007s
I20260924 11:47:16.667279 2896 heartbeater.cc:503] Master 127.1.210.62:43861 was elected leader, sending a full tablet report...
I20260924 11:47:16.668761 2915 ts_tablet_manager.cc:1428] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260924 11:47:16.671729 2916 raft_consensus.cc:348] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.672154 2916 raft_consensus.cc:374] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:16.672263 2916 raft_consensus.cc:729] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1371be9313dd447c8664887c913ab856, State: Initialized, Role: FOLLOWER
I20260924 11:47:16.672853 2916 consensus_queue.cc:260] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.674584 2763 heartbeater.cc:503] Master 127.1.210.62:43861 was elected leader, sending a full tablet report...
I20260924 11:47:16.675042 2916 ts_tablet_manager.cc:1428] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260924 11:47:16.787535 2897 tablet.cc:2369] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:47:16.815442 2764 tablet.cc:2369] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:47:16.843493 2630 tablet.cc:2369] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:47:16.849666 2921 raft_consensus.cc:482] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:16.849947 2921 raft_consensus.cc:504] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.851771 2921 leader_election.cc:288] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92894b8a823344b5b27faa2a67e971af (127.1.210.3:43671), 1371be9313dd447c8664887c913ab856 (127.1.210.2:36371)
I20260924 11:47:16.859165 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.852233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:59259 (local address 127.1.210.3:43671)
0924 11:47:16.852603 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.852610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.853586 (+ 976us) server_negotiation.cc:407] Connection header received
0924 11:47:16.853765 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.853770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.853847 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.853946 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:16.854619 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.855370 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.857494 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.858248 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.858347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.858475 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.858795 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.858916 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260924 11:47:16.859184 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.852364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:39213 (local address 127.1.210.2:36371)
0924 11:47:16.852560 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.852566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.853338 (+ 772us) server_negotiation.cc:407] Connection header received
0924 11:47:16.853544 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.853551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.853631 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.853721 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:16.854306 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.855209 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.857539 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.858277 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.858437 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.858529 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.858827 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.858942 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:47:16.859280 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.852485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43671 (local address 127.1.210.1:59259)
0924 11:47:16.853484 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.853502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.853620 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.853978 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.853981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.854000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:16.854422 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.854434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.855511 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.855518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.857353 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.857364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.858475 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.858481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.858639 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.858662 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.858733 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.858877 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":1,"thread_start_us":608,"threads_started":1}
I20260924 11:47:16.859326 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.852189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:36371 (local address 127.1.210.1:39213)
0924 11:47:16.853212 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:16.853231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:16.853386 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:16.853747 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:16.853751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:16.853777 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:16.854160 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.854170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.855345 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.855350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.857362 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:16.857379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.858526 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.858534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:16.858660 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.858681 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:47:16.858788 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 11:47:16.858908 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":62,"thread_start_us":582,"threads_started":1}
I20260924 11:47:16.860329 2718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c5fc320e3f4d238b57a3b45d4e7a74" candidate_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1371be9313dd447c8664887c913ab856" is_pre_election: true
I20260924 11:47:16.860523 2851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c5fc320e3f4d238b57a3b45d4e7a74" candidate_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92894b8a823344b5b27faa2a67e971af" is_pre_election: true
I20260924 11:47:16.860853 2718 raft_consensus.cc:2454] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3858ce0362e47b4b9728e8d4d28f65a in term 0.
I20260924 11:47:16.861050 2851 raft_consensus.cc:2454] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3858ce0362e47b4b9728e8d4d28f65a in term 0.
I20260924 11:47:16.861593 2517 leader_election.cc:302] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1371be9313dd447c8664887c913ab856, a3858ce0362e47b4b9728e8d4d28f65a; no voters:
I20260924 11:47:16.862020 2921 raft_consensus.cc:2781] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:16.862179 2921 raft_consensus.cc:482] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:16.862282 2921 raft_consensus.cc:3037] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:16.864400 2921 raft_consensus.cc:504] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.865029 2921 leader_election.cc:288] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [CANDIDATE]: Term 1 election: Requested vote from peers 92894b8a823344b5b27faa2a67e971af (127.1.210.3:43671), 1371be9313dd447c8664887c913ab856 (127.1.210.2:36371)
I20260924 11:47:16.865383 2851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c5fc320e3f4d238b57a3b45d4e7a74" candidate_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92894b8a823344b5b27faa2a67e971af"
I20260924 11:47:16.865527 2718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c5fc320e3f4d238b57a3b45d4e7a74" candidate_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1371be9313dd447c8664887c913ab856"
I20260924 11:47:16.865568 2851 raft_consensus.cc:3037] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:16.865746 2718 raft_consensus.cc:3037] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:16.868099 2718 raft_consensus.cc:2454] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 1371be9313dd447c8664887c913ab856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3858ce0362e47b4b9728e8d4d28f65a in term 1.
I20260924 11:47:16.868439 2851 raft_consensus.cc:2454] T b1c5fc320e3f4d238b57a3b45d4e7a74 P 92894b8a823344b5b27faa2a67e971af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3858ce0362e47b4b9728e8d4d28f65a in term 1.
I20260924 11:47:16.868510 2517 leader_election.cc:302] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1371be9313dd447c8664887c913ab856, a3858ce0362e47b4b9728e8d4d28f65a; no voters:
I20260924 11:47:16.868854 2921 raft_consensus.cc:2781] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:16.869333 2921 raft_consensus.cc:686] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [term 1 LEADER]: Becoming Leader. State: Replica: a3858ce0362e47b4b9728e8d4d28f65a, State: Running, Role: LEADER
I20260924 11:47:16.869796 2921 consensus_queue.cc:237] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } }
I20260924 11:47:16.874632 2440 catalog_manager.cc:5495] T b1c5fc320e3f4d238b57a3b45d4e7a74 P a3858ce0362e47b4b9728e8d4d28f65a reported cstate change: term changed from 0 to 1, leader changed from <none> to a3858ce0362e47b4b9728e8d4d28f65a (127.1.210.1). New cstate: current_term: 1 leader_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3858ce0362e47b4b9728e8d4d28f65a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92894b8a823344b5b27faa2a67e971af" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1371be9313dd447c8664887c913ab856" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36371 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:16.926483 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.911981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.1.210.1:42599)
0924 11:47:16.912166 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.912170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.912667 (+ 497us) server_negotiation.cc:407] Connection header received
0924 11:47:16.912885 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.912889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.912949 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.913049 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:16.914490 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.915257 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.916495 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.916905 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.917038 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.921322 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:16.921337 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:16.921352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:16.921381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:16.923812 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.924454 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.924461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.924464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.924604 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.924968 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.924975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.924978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.925368 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:16.925668 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.925932 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.926152 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:16.972846 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:16.959237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.1.210.1:42599)
0924 11:47:16.959442 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:16.959451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:16.959750 (+ 299us) server_negotiation.cc:407] Connection header received
0924 11:47:16.959965 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:16.959969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:16.960031 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:16.960137 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:16.961493 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.962330 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.963609 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:16.964019 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:16.964154 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:16.968363 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:16.968368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:16.968371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:16.968402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:16.970663 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.971287 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.971290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.971293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.971407 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:16.971760 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:16.971763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:16.971765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:16.972064 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:16.972277 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:16.972451 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 11:47:16.972605 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 11:47:16 dist-test-slave-dd32 krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:47:17 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:17.016181 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:17.002432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.1.210.1:42599)
0924 11:47:17.002664 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:17.002671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:17.002898 (+ 227us) server_negotiation.cc:407] Connection header received
0924 11:47:17.003042 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:17.003046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:17.003123 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:17.003206 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:17.004484 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.005310 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.006463 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.006967 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.007059 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:17.010755 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:17.010759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:17.010762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:17.010789 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:17.013412 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.014100 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.014103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.014105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.014221 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.014594 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.014597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.014599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.014830 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:17.015002 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:17.015755 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 11:47:17.015896 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 11:47:17.017956 2544 tablet_service.cc:2908] Rejecting scan request for tablet b1c5fc320e3f4d238b57a3b45d4e7a74: Uninitialized: clean time has not yet been initialized
I20260924 11:47:17.020780 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2501
I20260924 11:47:17.032855 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:17.021638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.1.210.1:42599)
0924 11:47:17.021831 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:17.021838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:17.022324 (+ 486us) server_negotiation.cc:407] Connection header received
0924 11:47:17.022511 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:17.022520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:17.022583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:17.022682 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:17.023764 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.024463 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.025713 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.026093 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.026216 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:17.028301 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:17.028306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:17.028309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:17.028335 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:17.030570 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.031209 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.031213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.031215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.031330 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.031636 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.031639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.031641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.031934 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:17.032092 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:17.032335 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 11:47:17.032499 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:47:17.033468 2624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:17.123957 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2501
I20260924 11:47:17.140770 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2635
Sep 24 11:47:17 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:17.154635 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:17.141628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.1.210.2:36371)
0924 11:47:17.141816 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:17.141820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:17.141837 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:47:17.141892 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:17.141896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:17.141954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:17.142035 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:17.143116 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.143863 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.145050 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.145508 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.145599 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:17.149380 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:17.149390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:17.149405 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:17.149437 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:17.152078 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.152718 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.152725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.152728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.152851 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.153210 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.153213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.153215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.153593 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:17.153818 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:17.154075 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 11:47:17.154279 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:47:17.155279 2758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:17.243093 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2635
I20260924 11:47:17.259231 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2768
Sep 24 11:47:17 dist-test-slave-dd32 krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:17.272980 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:17.259970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.1.210.3:43671)
0924 11:47:17.260125 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:17.260129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:17.260148 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:47:17.260267 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:17.260272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:17.260333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:17.260419 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:17.261650 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.262446 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.263690 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:17.264056 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:17.264145 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:17.267941 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:17.267954 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:17.267970 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:17.268004 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:17.270777 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.271292 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.271298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.271300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.271496 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:17.271817 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:17.271820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:17.271822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:17.272141 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:17.272340 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:17.272575 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 11:47:17.272722 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:47:17.273643 2891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:17.358867 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2768
I20260924 11:47:17.374939 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2408
I20260924 11:47:17.376048 2470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:17.482290 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2408
2026-09-24T11:47:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4128 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:17 dist-test-slave-dd32 krb5kdc[2944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:17 dist-test-slave-dd32 krb5kdc[2944](info): set up 2 sockets
Sep 24 11:47:17 dist-test-slave-dd32 krb5kdc[2944](info): commencing operation
krb5kdc: starting...
W20260924 11:47:18.564980 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:18 dist-test-slave-dd32 krb5kdc[2944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:18.740896 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42657
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:19.055127 2957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:19.055455 2957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:19.055613 2957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:19.062693 2957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:19.062839 2957 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:47:19.062938 2957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:19.063048 2957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:19.063139 2957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:19.063225 2957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:19.072908 2957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:42657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:19.074183 2957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:19.075555 2957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:19.087397 2962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:19.087422 2965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:19.087356 2963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:19.087996 2957 server_base.cc:1034] running on GCE node
I20260924 11:47:19.088728 2957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:19.090137 2957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:19.091377 2957 hybrid_clock.cc:648] HybridClock initialized: now 1790250439091313 us; error 42 us; skew 500 ppm
I20260924 11:47:19.091697 2957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:19 dist-test-slave-dd32 krb5kdc[2944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250439, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:19.098729 2957 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:19.100155 2957 webserver.cc:460] Webserver started at http://127.1.210.62:46057/ using document root <none> and password file <none>
I20260924 11:47:19.100843 2957 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:19.100992 2957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:19.101351 2957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:19.104194 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "db7d3aad44f04930a87125b1ee3b0020"
format_stamp: "Formatted at 2026-09-24 11:47:19 on dist-test-slave-dd32"
server_key: "65cc3dd2e7fe033b5b3f5bb1f8ab3974"
server_key_iv: "79932dd984de87461cb4e1465a4327d2"
server_key_version: "encryptionkey@0"
I20260924 11:47:19.105005 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "db7d3aad44f04930a87125b1ee3b0020"
format_stamp: "Formatted at 2026-09-24 11:47:19 on dist-test-slave-dd32"
server_key: "65cc3dd2e7fe033b5b3f5bb1f8ab3974"
server_key_iv: "79932dd984de87461cb4e1465a4327d2"
server_key_version: "encryptionkey@0"
I20260924 11:47:19.110069 2957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:47:19.113291 2972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:19.114451 2957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:47:19.114619 2957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "db7d3aad44f04930a87125b1ee3b0020"
format_stamp: "Formatted at 2026-09-24 11:47:19 on dist-test-slave-dd32"
server_key: "65cc3dd2e7fe033b5b3f5bb1f8ab3974"
server_key_iv: "79932dd984de87461cb4e1465a4327d2"
server_key_version: "encryptionkey@0"
I20260924 11:47:19.114817 2957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:19.130386 2957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:19.134169 2957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:19.134446 2957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:19.155496 2957 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:42657
I20260924 11:47:19.155522 3033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:42657 every 8 connection(s)
I20260924 11:47:19.157018 2957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:19.160593 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:19.166983 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 2957
I20260924 11:47:19.167212 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:19.167826 1864 external_mini_cluster.cc:1428] Setting key 4fe617f8cdd429117115719bd281135e
I20260924 11:47:19.171404 3034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: Bootstrap starting.
I20260924 11:47:19.175354 3034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:19.176398 3034 log.cc:824] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:19.179033 3034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: No bootstrap required, opened a new log
Sep 24 11:47:19 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:19.186601 3034 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7d3aad44f04930a87125b1ee3b0020" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42657 } }
I20260924 11:47:19.186985 3034 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:19.187104 3034 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7d3aad44f04930a87125b1ee3b0020, State: Initialized, Role: FOLLOWER
I20260924 11:47:19.187721 3034 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7d3aad44f04930a87125b1ee3b0020" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42657 } }
I20260924 11:47:19.187983 3034 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:19.188117 3034 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:19.188354 3034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:19.189503 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:19.169730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.1.210.62:42657)
0924 11:47:19.170490 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:19.170501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:19.170541 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:47:19.172547 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:19.172569 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:19.172958 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:19.173356 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:19.175071 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.176407 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.177692 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.178335 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.178537 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:19.182376 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:19.182384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:19.182397 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:19.182430 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:19.185477 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:19.186127 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:19.186133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:19.186139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:19.186296 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:19.186664 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:19.186673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:19.186675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:19.187268 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:19.187580 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:19.187988 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 11:47:19.188403 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":355,"threads_started":1}
I20260924 11:47:19.190766 3034 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7d3aad44f04930a87125b1ee3b0020" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42657 } }
I20260924 11:47:19.191253 3034 leader_election.cc:302] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7d3aad44f04930a87125b1ee3b0020; no voters:
I20260924 11:47:19.191898 3034 leader_election.cc:288] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:19.192015 3039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:19.192767 3039 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [term 1 LEADER]: Becoming Leader. State: Replica: db7d3aad44f04930a87125b1ee3b0020, State: Running, Role: LEADER
I20260924 11:47:19.193377 3039 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7d3aad44f04930a87125b1ee3b0020" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42657 } }
I20260924 11:47:19.194083 3034 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:19.196122 3040 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7d3aad44f04930a87125b1ee3b0020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7d3aad44f04930a87125b1ee3b0020" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42657 } } }
I20260924 11:47:19.196481 3040 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:19.197028 3041 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7d3aad44f04930a87125b1ee3b0020. Latest consensus state: current_term: 1 leader_uuid: "db7d3aad44f04930a87125b1ee3b0020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7d3aad44f04930a87125b1ee3b0020" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42657 } } }
I20260924 11:47:19.197330 3041 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:19.201858 3048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:19.206633 3048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:19.214567 3048 catalog_manager.cc:1347] Generated new cluster ID: a0f5c6f5b81a4f69a61cc60e3f87c002
I20260924 11:47:19.214677 3048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:19.225900 3048 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:19.227397 3048 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:19.236872 3048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: Generated new TSK 0
I20260924 11:47:19.237885 3048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:19.285928 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42657
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:19.598903 3060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:19.599216 3060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:19.599350 3060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:19.606534 3060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:19.606640 3060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:19.606731 3060 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:47:19.606813 3060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:19.616221 3060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:19.617443 3060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:19.618875 3060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:19.631728 3065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:19.631851 3066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:19.631938 3068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:19.632733 3060 server_base.cc:1034] running on GCE node
I20260924 11:47:19.633344 3060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:19.634153 3060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:19.635378 3060 hybrid_clock.cc:648] HybridClock initialized: now 1790250439635307 us; error 49 us; skew 500 ppm
I20260924 11:47:19.635653 3060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:19 dist-test-slave-dd32 krb5kdc[2944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250439, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:19.638759 3060 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:19.640161 3060 webserver.cc:460] Webserver started at http://127.1.210.1:45729/ using document root <none> and password file <none>
I20260924 11:47:19.640798 3060 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:19.640955 3060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:19.641309 3060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:19.644008 3060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "81a8291168d1478690a617495105a2fb"
format_stamp: "Formatted at 2026-09-24 11:47:19 on dist-test-slave-dd32"
server_key: "e3c34fa79dca0514a341184db66530d9"
server_key_iv: "123808d14163b25b80d336ce879e2c22"
server_key_version: "encryptionkey@0"
I20260924 11:47:19.644768 3060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "81a8291168d1478690a617495105a2fb"
format_stamp: "Formatted at 2026-09-24 11:47:19 on dist-test-slave-dd32"
server_key: "e3c34fa79dca0514a341184db66530d9"
server_key_iv: "123808d14163b25b80d336ce879e2c22"
server_key_version: "encryptionkey@0"
I20260924 11:47:19.649542 3060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:47:19.652791 3075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:19.654016 3060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:19.654176 3060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "81a8291168d1478690a617495105a2fb"
format_stamp: "Formatted at 2026-09-24 11:47:19 on dist-test-slave-dd32"
server_key: "e3c34fa79dca0514a341184db66530d9"
server_key_iv: "123808d14163b25b80d336ce879e2c22"
server_key_version: "encryptionkey@0"
I20260924 11:47:19.654374 3060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:19.686221 3060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:19.690999 3060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:19.691274 3060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:19.694049 3060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:19.694159 3060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:19.694377 3060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:19.694450 3060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:47:19 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250439, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:19.716697 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:19.695172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.1.210.1:39249)
0924 11:47:19.695973 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:19.696012 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:19.697955 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:19.698670 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:19.698678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:19.699625 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:19.700585 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:19.700610 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.702088 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.702095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:19.704324 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:19.704335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.705131 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.705140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:19.705406 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:19.706283 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:19.706320 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:19.708924 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:19.712049 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:19.712061 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:19.712079 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:19.712682 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:19.713180 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:19.713184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:19.713189 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:19.713409 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:19.714131 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:19.714140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:19.714552 (+ 412us) client_negotiation.cc:769] Sending connection context
0924 11:47:19.714900 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 11:47:19.715411 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":343,"threads_started":1}
I20260924 11:47:19.716943 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:19.694458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:39249 (local address 127.1.210.62:42657)
0924 11:47:19.694997 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:19.695005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:19.696133 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 11:47:19.698224 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:19.698229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:19.698309 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:19.698412 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:19.700842 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.701922 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.704478 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.704893 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.705026 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:19.709146 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:19.709151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:19.709154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:19.709184 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:19.711906 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:19.712818 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:19.712828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:19.712834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:19.713022 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:19.713526 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:19.713534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:19.713540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:19.713965 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:19.714133 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:19.716123 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0924 11:47:19.716431 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":314,"threads_started":1}
I20260924 11:47:19.740092 3060 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:41761
I20260924 11:47:19.740144 3195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:41761 every 8 connection(s)
I20260924 11:47:19.740617 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:19.724837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:38385 (local address 127.1.210.62:42657)
0924 11:47:19.724979 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:19.724983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:19.725939 (+ 956us) server_negotiation.cc:407] Connection header received
0924 11:47:19.726137 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:19.726142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:19.726214 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:19.726311 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:19.727654 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.728645 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.730855 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.731271 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.731440 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:19.733568 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:19.733577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:19.733584 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:19.733628 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:19.737390 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:19.738332 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:19.738336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:19.738338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:19.738453 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:19.738937 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:19.738942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:19.738945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:19.739302 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:19.739523 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:19.740092 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 11:47:19.740321 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:47:19.740724 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:19.724896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.1.210.1:38385)
0924 11:47:19.725818 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:19.725840 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:19.725990 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:19.726342 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:19.726349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:19.727067 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:19.727517 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:19.727534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.728838 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.728841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:19.730704 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:19.730715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:19.731496 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:19.731504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:19.731670 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:19.732303 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:19.732321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:19.733382 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:19.737541 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:19.737550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:19.737557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:19.738206 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:19.738583 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:19.738588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:19.738592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:19.738798 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:19.739504 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:19.739515 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:19.739768 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 11:47:19.739955 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 11:47:19.740283 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":533,"threads_started":1}
I20260924 11:47:19.741691 3060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:19.743782 3196 heartbeater.cc:347] Connected to a master server at 127.1.210.62:42657
I20260924 11:47:19.744015 3196 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:19.744644 3196 heartbeater.cc:511] Master 127.1.210.62:42657 requested a full tablet report, sending...
I20260924 11:47:19.746490 2989 ts_manager.cc:194] Registered new tserver with Master: 81a8291168d1478690a617495105a2fb (127.1.210.1:41761)
I20260924 11:47:19.747844 2989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:39249
I20260924 11:47:19.751641 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3060
I20260924 11:47:19.751798 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:19.752369 1864 external_mini_cluster.cc:1428] Setting key c9e9658db7e02f3e896b32679c4f1af3
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:19.790508 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42657
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:20.099905 3200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:20.100207 3200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:20.100373 3200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:20.107436 3200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:20.107573 3200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:20.107721 3200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:47:20.107863 3200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:20.117143 3200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:20.118507 3200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:20.119969 3200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:20.132756 3205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:20.132705 3206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:20.132911 3208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:20.133651 3200 server_base.cc:1034] running on GCE node
I20260924 11:47:20.134204 3200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:20.135084 3200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:20.136328 3200 hybrid_clock.cc:648] HybridClock initialized: now 1790250440136251 us; error 50 us; skew 500 ppm
I20260924 11:47:20.136616 3200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:20 dist-test-slave-dd32 krb5kdc[2944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:20.139645 3200 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:20.141012 3200 webserver.cc:460] Webserver started at http://127.1.210.2:37193/ using document root <none> and password file <none>
I20260924 11:47:20.141711 3200 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:20.141837 3200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:20.142155 3200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:20.144938 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "a0dcbf8828244684ab7b9be994b0d5d7"
format_stamp: "Formatted at 2026-09-24 11:47:20 on dist-test-slave-dd32"
server_key: "0f934f86362f6819612cbf3f1fb4351e"
server_key_iv: "3b7cea41f02cd54bfc6656d0a35cfec9"
server_key_version: "encryptionkey@0"
I20260924 11:47:20.145773 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "a0dcbf8828244684ab7b9be994b0d5d7"
format_stamp: "Formatted at 2026-09-24 11:47:20 on dist-test-slave-dd32"
server_key: "0f934f86362f6819612cbf3f1fb4351e"
server_key_iv: "3b7cea41f02cd54bfc6656d0a35cfec9"
server_key_version: "encryptionkey@0"
I20260924 11:47:20.150729 3200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 11:47:20.154078 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:20.155656 3200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:47:20.155807 3200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "a0dcbf8828244684ab7b9be994b0d5d7"
format_stamp: "Formatted at 2026-09-24 11:47:20 on dist-test-slave-dd32"
server_key: "0f934f86362f6819612cbf3f1fb4351e"
server_key_iv: "3b7cea41f02cd54bfc6656d0a35cfec9"
server_key_version: "encryptionkey@0"
I20260924 11:47:20.156020 3200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:20.180722 3200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:20.184518 3200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:20.184727 3200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:20.187342 3200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:20.187440 3200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:20.187597 3200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:20.187705 3200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 11:47:20 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:20.212579 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.187831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:37729 (local address 127.1.210.62:42657)
0924 11:47:20.188019 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.188027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.189456 (+ 1429us) server_negotiation.cc:407] Connection header received
0924 11:47:20.191526 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.191532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.191654 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.191763 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:20.193975 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.195335 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.198770 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.199662 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.199824 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.204646 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:20.204651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:20.204655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:20.204685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:20.208942 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.209705 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.209713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.209719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.209869 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.210281 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.210287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.210290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.210687 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:20.210875 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.211855 (+ 980us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.212116 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:47:20.214066 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.188450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.1.210.2:37729)
0924 11:47:20.189299 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.189342 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.191202 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.192041 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.192057 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.192819 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:20.193679 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.193701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.195561 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.195570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.198522 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.198546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.199886 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.199890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.200075 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.200746 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:20.200771 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:20.204351 (+ 3580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:20.209135 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.209142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.209155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.209574 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.209967 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.209970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.209972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.210142 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.210849 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:20.210861 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:20.211353 (+ 492us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.211713 (+ 360us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.212368 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":349,"threads_started":1}
I20260924 11:47:20.232468 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.220438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:44125 (local address 127.1.210.62:42657)
0924 11:47:20.220765 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.220773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.220797 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:47:20.220858 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.220866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.220973 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.221131 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:20.222445 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.223238 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.224530 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.224942 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.225074 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.226718 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:20.226722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:20.226726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:20.226753 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:20.229711 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.230588 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.230592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.230594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.230727 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.231136 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.231143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.231146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.231446 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:20.231590 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.231949 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.232145 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260924 11:47:20.232599 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.219917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.1.210.2:44125)
0924 11:47:20.220616 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.220631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.220724 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.221161 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.221168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.221841 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:20.222297 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.222307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.223381 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.223388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.224388 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.224398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.225148 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.225156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.225319 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.225894 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:20.225909 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:20.226535 (+ 626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:20.229840 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.229845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.229849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.230420 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.230880 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.230886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.230895 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.231002 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.231559 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:20.231563 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:20.231723 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.231866 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.232163 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":393,"threads_started":1}
I20260924 11:47:20.247318 3200 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:38241
I20260924 11:47:20.247412 3334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:38241 every 8 connection(s)
I20260924 11:47:20.248504 3200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:20.250451 3335 heartbeater.cc:347] Connected to a master server at 127.1.210.62:42657
I20260924 11:47:20.250665 3335 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:20.251251 3335 heartbeater.cc:511] Master 127.1.210.62:42657 requested a full tablet report, sending...
I20260924 11:47:20.252494 2989 ts_manager.cc:194] Registered new tserver with Master: a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241)
I20260924 11:47:20.253283 2989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:37729
I20260924 11:47:20.257743 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3200
I20260924 11:47:20.257900 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:20.258306 1864 external_mini_cluster.cc:1428] Setting key 25b965ac1c0542334b069515359e1f34
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:20.293859 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42657
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:20.602401 3339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:20.602697 3339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:20.602869 3339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:20.609866 3339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:20.610005 3339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:20.610162 3339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:47:20.610298 3339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:20.620003 3339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:20.621312 3339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:20.622622 3339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:20.635174 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:20.635205 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:20.635228 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:20.635792 3339 server_base.cc:1034] running on GCE node
I20260924 11:47:20.636377 3339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:20.637272 3339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:20.638517 3339 hybrid_clock.cc:648] HybridClock initialized: now 1790250440638460 us; error 36 us; skew 500 ppm
I20260924 11:47:20.638831 3339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:20 dist-test-slave-dd32 krb5kdc[2944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:20.641834 3339 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:20.643085 3339 webserver.cc:460] Webserver started at http://127.1.210.3:37519/ using document root <none> and password file <none>
I20260924 11:47:20.643723 3339 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:20.643855 3339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:20.644164 3339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:20.646886 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "aea86f9982ff4290a7310cae63e2b4f3"
format_stamp: "Formatted at 2026-09-24 11:47:20 on dist-test-slave-dd32"
server_key: "e9513a076ba16b31fd8a19f901e41c71"
server_key_iv: "9f9f192b138234607f5767484dbfb849"
server_key_version: "encryptionkey@0"
I20260924 11:47:20.647642 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "aea86f9982ff4290a7310cae63e2b4f3"
format_stamp: "Formatted at 2026-09-24 11:47:20 on dist-test-slave-dd32"
server_key: "e9513a076ba16b31fd8a19f901e41c71"
server_key_iv: "9f9f192b138234607f5767484dbfb849"
server_key_version: "encryptionkey@0"
I20260924 11:47:20.652448 3339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 11:47:20.655845 3354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:20.657169 3339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:47:20.657334 3339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "aea86f9982ff4290a7310cae63e2b4f3"
format_stamp: "Formatted at 2026-09-24 11:47:20 on dist-test-slave-dd32"
server_key: "e9513a076ba16b31fd8a19f901e41c71"
server_key_iv: "9f9f192b138234607f5767484dbfb849"
server_key_version: "encryptionkey@0"
I20260924 11:47:20.657584 3339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:20.683607 3339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:20.687398 3339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:20.687666 3339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:20.690304 3339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:20.690420 3339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:20.690610 3339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:20.690683 3339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:47:20 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:20.713259 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.690647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:45627 (local address 127.1.210.62:42657)
0924 11:47:20.690875 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.690881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.692222 (+ 1341us) server_negotiation.cc:407] Connection header received
0924 11:47:20.694357 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.694364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.694460 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.694592 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:20.697024 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.698193 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.700879 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.701384 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.701522 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.705235 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:20.705243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:20.705250 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:20.705298 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:20.709741 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.710667 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.710675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.710679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.710866 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.711422 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.711431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.711434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.711711 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:20.711865 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.712586 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.712859 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:47:20.714754 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.691191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.1.210.3:45627)
0924 11:47:20.692035 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.692088 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.694005 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.694906 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.694917 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.695785 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:20.696839 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.696866 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.698414 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.698426 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.700723 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.700737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.701606 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.701612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.701840 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.702505 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:20.702526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:20.704974 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:20.709924 (+ 4950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.709935 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.709961 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.710519 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.711029 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.711039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.711046 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.711286 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.711846 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:20.711854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:20.712206 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.712465 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.712994 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":365,"threads_started":1}
I20260924 11:47:20.742727 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.727892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:33907 (local address 127.1.210.62:42657)
0924 11:47:20.728092 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.728099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.728562 (+ 463us) server_negotiation.cc:407] Connection header received
0924 11:47:20.728714 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.728720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.728798 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.728904 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:20.730360 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.731568 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.733546 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.734125 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.734266 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.736038 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:20.736042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:20.736045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:20.736074 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:20.738911 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.740005 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.740013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.740020 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.740199 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.740742 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.740750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.740756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.741216 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:20.741378 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.742340 (+ 962us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.742499 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:47:20.743361 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.727901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.1.210.3:33907)
0924 11:47:20.728446 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.728465 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.728575 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.728931 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.728936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.729630 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:20.730161 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.730183 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.731704 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.731707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.733367 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.733383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.734311 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.734317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.734491 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.735140 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:20.735156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:20.735855 (+ 699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:20.739056 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.739063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.739069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.739753 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.740392 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:20.740399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:20.740406 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:20.740587 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:20.741445 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:20.741453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:20.741690 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.741869 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.743010 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":289,"threads_started":1}
I20260924 11:47:20.748193 3339 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:38325
I20260924 11:47:20.748231 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:38325 every 8 connection(s)
I20260924 11:47:20.749379 3339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:20.750336 3196 heartbeater.cc:503] Master 127.1.210.62:42657 was elected leader, sending a full tablet report...
I20260924 11:47:20.750411 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3339
I20260924 11:47:20.750586 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:20.751088 1864 external_mini_cluster.cc:1428] Setting key c37b102d418b411bd7a033d32bce365b
I20260924 11:47:20.751519 3474 heartbeater.cc:347] Connected to a master server at 127.1.210.62:42657
I20260924 11:47:20.751770 3474 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:20.754082 3474 heartbeater.cc:511] Master 127.1.210.62:42657 requested a full tablet report, sending...
I20260924 11:47:20.755283 2989 ts_manager.cc:194] Registered new tserver with Master: aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:20.755988 2989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:45627
I20260924 11:47:20.765996 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:47:20 dist-test-slave-dd32 krb5kdc[2944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:47:20 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:20.809628 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.795958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.1.210.62:42657)
0924 11:47:20.796156 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.796165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.796531 (+ 366us) server_negotiation.cc:407] Connection header received
0924 11:47:20.796743 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.796747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.796812 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.796925 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:20.798413 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.799138 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.800394 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.800793 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.800953 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.804737 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:20.804742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:20.804745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:20.804777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:20.807073 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.807709 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.807720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.807722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.807837 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:20.808211 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:20.808216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:20.808219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:20.808477 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:20.808730 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.809008 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.809251 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:47:20.813321 2989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:20.820353 2989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:20.854561 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.832363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.1.210.2:38241)
0924 11:47:20.845962 (+ 13599us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.845976 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.846019 (+ 43us) server_negotiation.cc:407] Connection header received
0924 11:47:20.846150 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.846164 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.846509 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.846783 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:20.847760 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.849225 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.851992 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.853378 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.853515 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.853724 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.854072 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.854240 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13377,"thread_start_us":13208,"threads_started":1}
I20260924 11:47:20.854704 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.831457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:38241 (local address 127.0.0.1:35930)
0924 11:47:20.844559 (+ 13102us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.844583 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.844726 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.846838 (+ 2112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.846849 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.846889 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:20.847476 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.847494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.849358 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.849363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.851801 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.851820 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.853601 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.853608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.853819 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.853874 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.854016 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.854197 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12921,"mutex_wait_us":3,"thread_start_us":12767,"threads_started":1}
I20260924 11:47:20.856509 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.831246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:41761 (local address 127.0.0.1:47608)
0924 11:47:20.840729 (+ 9483us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.840778 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.841012 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.848680 (+ 7668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.848685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.848723 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:20.849453 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.849469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.850849 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.850855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.852875 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.852887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.855157 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.855165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.855711 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.855803 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.855902 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.856027 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9158,"mutex_wait_us":159,"spinlock_wait_cycles":896,"thread_start_us":15965,"threads_started":1}
I20260924 11:47:20.856881 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.831436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.1.210.1:41761)
0924 11:47:20.848184 (+ 16748us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.848193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.848218 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:47:20.848309 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.848316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.848493 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.848682 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:20.849727 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.850700 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.853038 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.854405 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.855230 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.855491 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.856343 (+ 852us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.856540 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16600,"thread_start_us":17368,"threads_started":1}
I20260924 11:47:20.856576 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.831858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38325 (local address 127.0.0.1:52068)
0924 11:47:20.847483 (+ 15625us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:20.847503 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:20.847634 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:20.848680 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:20.848685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:20.848731 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:20.849452 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.849469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.851474 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.851479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.853613 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:20.853625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.855704 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.855712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:20.855942 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.855974 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:47:20.856054 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 11:47:20.856174 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15505,"mutex_wait_us":76,"spinlock_wait_cycles":7680,"thread_start_us":8339,"threads_started":1}
I20260924 11:47:20.858048 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:20.832951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.1.210.3:38325)
0924 11:47:20.840547 (+ 7596us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:20.840556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:20.847776 (+ 7220us) server_negotiation.cc:407] Connection header received
0924 11:47:20.847895 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:20.847905 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:20.848131 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:20.848327 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:20.849700 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.851298 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.853791 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:20.855159 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:20.855300 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:20.855511 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:20.856719 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0924 11:47:20.856880 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7433,"thread_start_us":7951,"threads_started":1}
I20260924 11:47:20.858707 3265 tablet_service.cc:1467] Processing CreateTablet for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f (DEFAULT_TABLE table=test-table [id=9383f971b3d74f489ec2e83c0d36a289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:20.859973 3265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:20.860248 3126 tablet_service.cc:1467] Processing CreateTablet for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f (DEFAULT_TABLE table=test-table [id=9383f971b3d74f489ec2e83c0d36a289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:20.861749 3126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:20.861354 3404 tablet_service.cc:1467] Processing CreateTablet for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f (DEFAULT_TABLE table=test-table [id=9383f971b3d74f489ec2e83c0d36a289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:20.862709 3404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:20.871093 3493 tablet_bootstrap.cc:492] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Bootstrap starting.
I20260924 11:47:20.874603 3493 tablet_bootstrap.cc:654] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:20.875649 3493 log.cc:824] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:20.875955 3494 tablet_bootstrap.cc:492] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Bootstrap starting.
I20260924 11:47:20.876897 3495 tablet_bootstrap.cc:492] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Bootstrap starting.
I20260924 11:47:20.879693 3495 tablet_bootstrap.cc:654] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:20.880004 3494 tablet_bootstrap.cc:654] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:20.880718 3495 log.cc:824] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:20.881428 3494 log.cc:824] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:20.883643 3493 tablet_bootstrap.cc:492] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: No bootstrap required, opened a new log
I20260924 11:47:20.883929 3493 ts_tablet_manager.cc:1397] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260924 11:47:20.884035 3495 tablet_bootstrap.cc:492] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: No bootstrap required, opened a new log
I20260924 11:47:20.884310 3495 ts_tablet_manager.cc:1397] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:47:20.884425 3494 tablet_bootstrap.cc:492] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: No bootstrap required, opened a new log
I20260924 11:47:20.884667 3494 ts_tablet_manager.cc:1397] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 11:47:20.891809 3493 raft_consensus.cc:348] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:20.892302 3493 raft_consensus.cc:374] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:20.892431 3493 raft_consensus.cc:729] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dcbf8828244684ab7b9be994b0d5d7, State: Initialized, Role: FOLLOWER
I20260924 11:47:20.893203 3493 consensus_queue.cc:260] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:20.894439 3335 heartbeater.cc:503] Master 127.1.210.62:42657 was elected leader, sending a full tablet report...
I20260924 11:47:20.894991 3493 ts_tablet_manager.cc:1428] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260924 11:47:20.895196 3494 raft_consensus.cc:348] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:20.895622 3494 raft_consensus.cc:374] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:20.895745 3494 raft_consensus.cc:729] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a8291168d1478690a617495105a2fb, State: Initialized, Role: FOLLOWER
I20260924 11:47:20.896287 3494 consensus_queue.cc:260] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:20.896144 3495 raft_consensus.cc:348] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:20.896535 3495 raft_consensus.cc:374] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:20.896661 3495 raft_consensus.cc:729] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea86f9982ff4290a7310cae63e2b4f3, State: Initialized, Role: FOLLOWER
I20260924 11:47:20.897290 3495 consensus_queue.cc:260] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:20.898053 3494 ts_tablet_manager.cc:1428] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260924 11:47:20.900795 3474 heartbeater.cc:503] Master 127.1.210.62:42657 was elected leader, sending a full tablet report...
I20260924 11:47:20.901460 3495 ts_tablet_manager.cc:1428] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
W20260924 11:47:20.994238 3197 tablet.cc:2369] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:47:21.000437 3336 tablet.cc:2369] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:47:21.000815 3475 tablet.cc:2369] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:47:21.310182 3499 raft_consensus.cc:482] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:21.310526 3499 raft_consensus.cc:504] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.312117 3499 leader_election.cc:288] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a8291168d1478690a617495105a2fb (127.1.210.1:41761), aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:21.313282 3501 raft_consensus.cc:482] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:21.313632 3501 raft_consensus.cc:504] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.316190 3501 leader_election.cc:288] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a8291168d1478690a617495105a2fb (127.1.210.1:41761), a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241)
I20260924 11:47:21.322801 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.312596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:60547 (local address 127.1.210.1:41761)
0924 11:47:21.312789 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.312796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.313574 (+ 778us) server_negotiation.cc:407] Connection header received
0924 11:47:21.313891 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.313900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.313994 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.314089 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:21.314705 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.315706 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.320376 (+ 4670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.321645 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.321746 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.321838 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.322433 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.322557 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:47:21.323352 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.312605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:36375 (local address 127.1.210.3:38325)
0924 11:47:21.312789 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.312796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.313919 (+ 1123us) server_negotiation.cc:407] Connection header received
0924 11:47:21.314153 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.314158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.314229 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.314285 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:21.314993 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.316130 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.320891 (+ 4761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.322177 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.322264 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.322381 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.322917 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.323096 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":67}
I20260924 11:47:21.323745 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.315912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:44073 (local address 127.1.210.2:38241)
0924 11:47:21.316052 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.316058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.316245 (+ 187us) server_negotiation.cc:407] Connection header received
0924 11:47:21.316536 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.316542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.316607 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.316669 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:21.317286 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.318789 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.321329 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.322522 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.322619 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.322711 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.323359 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.323482 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":30}
I20260924 11:47:21.324113 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.312654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:41761 (local address 127.1.210.2:60547)
0924 11:47:21.313462 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.313480 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.313711 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.314128 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.314133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.314158 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:21.314531 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.314548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.316845 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.316852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.320183 (+ 3331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.320198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.322039 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.322046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.322238 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.322269 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.323270 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.323422 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":8,"thread_start_us":451,"threads_started":1}
I20260924 11:47:21.324198 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.312654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38325 (local address 127.1.210.2:36375)
0924 11:47:21.313826 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.313844 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.314007 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.314435 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.314438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.314458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:21.314817 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.314828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.316642 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.316647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.320708 (+ 4061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.320722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.322567 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.322574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.322734 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.322764 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.323564 (+ 800us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.323693 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":24,"thread_start_us":863,"threads_started":1}
I20260924 11:47:21.324550 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.315596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:38241 (local address 127.1.210.3:44073)
0924 11:47:21.316150 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.316169 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.316378 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.316874 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.316881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.316906 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:21.317134 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.317142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.319055 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.319061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.321155 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.321169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.322849 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.322857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.323184 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.323225 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.324048 (+ 823us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.324191 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":25,"thread_start_us":819,"threads_started":1}
I20260924 11:47:21.325500 3290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "aea86f9982ff4290a7310cae63e2b4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" is_pre_election: true
I20260924 11:47:21.325865 3290 raft_consensus.cc:2454] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea86f9982ff4290a7310cae63e2b4f3 in term 0.
I20260924 11:47:21.325366 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea86f9982ff4290a7310cae63e2b4f3" is_pre_election: true
I20260924 11:47:21.325822 3150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a8291168d1478690a617495105a2fb" is_pre_election: true
I20260924 11:47:21.326246 3425 raft_consensus.cc:2454] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0dcbf8828244684ab7b9be994b0d5d7 in term 0.
I20260924 11:47:21.326318 3150 raft_consensus.cc:2454] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0dcbf8828244684ab7b9be994b0d5d7 in term 0.
I20260924 11:47:21.326543 3355 leader_election.cc:302] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0dcbf8828244684ab7b9be994b0d5d7, aea86f9982ff4290a7310cae63e2b4f3; no voters:
I20260924 11:47:21.326836 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.315610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:41761 (local address 127.1.210.3:44291)
0924 11:47:21.316629 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.316648 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.316732 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.319513 (+ 2781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.319518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.319548 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:21.319883 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.319893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.321480 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.321485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.323585 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.323598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.325991 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.325998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.326178 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.326218 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.326315 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.326447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":69,"thread_start_us":347,"threads_started":1}
I20260924 11:47:21.327099 3217 leader_election.cc:302] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a8291168d1478690a617495105a2fb, a0dcbf8828244684ab7b9be994b0d5d7; no voters:
I20260924 11:47:21.327175 3501 raft_consensus.cc:2781] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:21.327320 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.316623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:44291 (local address 127.1.210.1:41761)
0924 11:47:21.318937 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.318945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.318969 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:47:21.319024 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.319029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.319162 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.319263 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:21.320074 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.321323 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.323755 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.325124 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.325237 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.325819 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.326900 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.327024 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2180,"thread_start_us":1920,"threads_started":1}
I20260924 11:47:21.327292 3501 raft_consensus.cc:482] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:21.327397 3501 raft_consensus.cc:3037] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.327719 3499 raft_consensus.cc:2781] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:21.327878 3499 raft_consensus.cc:482] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:21.327842 3150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "aea86f9982ff4290a7310cae63e2b4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a8291168d1478690a617495105a2fb" is_pre_election: true
I20260924 11:47:21.327984 3499 raft_consensus.cc:3037] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.328032 3150 raft_consensus.cc:2454] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea86f9982ff4290a7310cae63e2b4f3 in term 0.
I20260924 11:47:21.330183 3501 raft_consensus.cc:504] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.330302 3499 raft_consensus.cc:504] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.330806 3501 leader_election.cc:288] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [CANDIDATE]: Term 1 election: Requested vote from peers 81a8291168d1478690a617495105a2fb (127.1.210.1:41761), a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241)
I20260924 11:47:21.330916 3499 leader_election.cc:288] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 81a8291168d1478690a617495105a2fb (127.1.210.1:41761), aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:21.331192 3150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "aea86f9982ff4290a7310cae63e2b4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a8291168d1478690a617495105a2fb"
I20260924 11:47:21.331301 3290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "aea86f9982ff4290a7310cae63e2b4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dcbf8828244684ab7b9be994b0d5d7"
I20260924 11:47:21.331372 3150 raft_consensus.cc:3037] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.331547 3290 raft_consensus.cc:2379] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aea86f9982ff4290a7310cae63e2b4f3 in current term 1: Already voted for candidate a0dcbf8828244684ab7b9be994b0d5d7 in this term.
I20260924 11:47:21.331632 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea86f9982ff4290a7310cae63e2b4f3"
I20260924 11:47:21.331856 3425 raft_consensus.cc:2379] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0dcbf8828244684ab7b9be994b0d5d7 in current term 1: Already voted for candidate aea86f9982ff4290a7310cae63e2b4f3 in this term.
I20260924 11:47:21.331761 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d3b0b4cfd04f2f8851b8971fc3c31f" candidate_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a8291168d1478690a617495105a2fb"
I20260924 11:47:21.332602 3217 leader_election.cc:302] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0dcbf8828244684ab7b9be994b0d5d7; no voters: 81a8291168d1478690a617495105a2fb, aea86f9982ff4290a7310cae63e2b4f3
I20260924 11:47:21.332880 3499 raft_consensus.cc:2726] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 11:47:21.334051 3150 raft_consensus.cc:2454] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea86f9982ff4290a7310cae63e2b4f3 in term 1.
I20260924 11:47:21.334370 3356 leader_election.cc:302] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81a8291168d1478690a617495105a2fb, aea86f9982ff4290a7310cae63e2b4f3; no voters: a0dcbf8828244684ab7b9be994b0d5d7
I20260924 11:47:21.334684 3501 raft_consensus.cc:2781] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:21.335047 3501 raft_consensus.cc:686] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 1 LEADER]: Becoming Leader. State: Replica: aea86f9982ff4290a7310cae63e2b4f3, State: Running, Role: LEADER
I20260924 11:47:21.335484 3501 consensus_queue.cc:237] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.340189 2988 catalog_manager.cc:5495] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "aea86f9982ff4290a7310cae63e2b4f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } health_report { overall_health: HEALTHY } } }
I20260924 11:47:21.421840 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.415828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.1.210.3:38325)
0924 11:47:21.416097 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.416105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.416503 (+ 398us) server_negotiation.cc:407] Connection header received
0924 11:47:21.416871 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.416875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.416954 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.417123 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:21.417622 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.418381 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.419797 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.420228 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.420385 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.420766 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:21.421349 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:21.421452 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.421557 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.421617 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:47:21.429226 3150 raft_consensus.cc:1261] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Refusing update from remote peer aea86f9982ff4290a7310cae63e2b4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:21.429244 3290 raft_consensus.cc:1261] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Refusing update from remote peer aea86f9982ff4290a7310cae63e2b4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:21.430066 3507 consensus_queue.cc:1035] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:21.430435 3501 consensus_queue.cc:1035] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:21.441388 3516 mvcc.cc:205] Tried to move back new op lower bound from 7332865808084697088 to 7332865807715254272. Current Snapshot: MvccSnapshot[applied={T|T < 7332865808084697088}]
I20260924 11:47:21.459021 2988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:21.459280 2988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:21.462807 2988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: Sending DeleteTablet for 3 replicas of tablet 68d3b0b4cfd04f2f8851b8971fc3c31f
I20260924 11:47:21.463855 3126 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:21 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:47608
I20260924 11:47:21.463997 3265 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:21 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:35930
I20260924 11:47:21.464883 3404 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d3b0b4cfd04f2f8851b8971fc3c31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:21 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:52068
I20260924 11:47:21.467753 2988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:21.469533 2988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:21.479115 3125 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1bc1469e844f02a228338d1acbfc4d (DEFAULT_TABLE table=test-table [id=3a5b544fd4c140d297decbd40be40caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:21.479206 3403 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1bc1469e844f02a228338d1acbfc4d (DEFAULT_TABLE table=test-table [id=3a5b544fd4c140d297decbd40be40caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:21.479699 3125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1bc1469e844f02a228338d1acbfc4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:21.479746 3403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1bc1469e844f02a228338d1acbfc4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:21.486538 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1bc1469e844f02a228338d1acbfc4d (DEFAULT_TABLE table=test-table [id=3a5b544fd4c140d297decbd40be40caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:21.487090 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1bc1469e844f02a228338d1acbfc4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:21.496636 3527 tablet_replica.cc:330] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: stopping tablet replica
I20260924 11:47:21.496697 3528 tablet_bootstrap.cc:492] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Bootstrap starting.
I20260924 11:47:21.498147 3530 tablet_bootstrap.cc:492] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Bootstrap starting.
I20260924 11:47:21.500061 3530 tablet_bootstrap.cc:654] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:21.497189 3527 raft_consensus.cc:2229] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:21.502763 3528 tablet_bootstrap.cc:654] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:21.504318 3529 tablet_bootstrap.cc:492] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Bootstrap starting.
I20260924 11:47:21.505432 3527 raft_consensus.cc:2258] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:21.506295 3529 tablet_bootstrap.cc:654] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:21.507084 3530 tablet_bootstrap.cc:492] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: No bootstrap required, opened a new log
I20260924 11:47:21.507282 3530 ts_tablet_manager.cc:1397] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260924 11:47:21.507550 3527 ts_tablet_manager.cc:1905] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:21.508248 3525 tablet_replica.cc:330] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: stopping tablet replica
I20260924 11:47:21.508662 3525 raft_consensus.cc:2229] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:21.509284 3525 raft_consensus.cc:2258] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:21.509616 3530 raft_consensus.cc:348] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.509943 3530 raft_consensus.cc:374] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:21.510097 3530 raft_consensus.cc:729] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a8291168d1478690a617495105a2fb, State: Initialized, Role: FOLLOWER
I20260924 11:47:21.510427 3530 consensus_queue.cc:260] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.511065 3530 ts_tablet_manager.cc:1428] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:21.512427 3529 tablet_bootstrap.cc:492] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: No bootstrap required, opened a new log
I20260924 11:47:21.512562 3529 ts_tablet_manager.cc:1397] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 11:47:21.514254 3525 ts_tablet_manager.cc:1905] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:21.516078 3529 raft_consensus.cc:348] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.516319 3529 raft_consensus.cc:374] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:21.516433 3529 raft_consensus.cc:729] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dcbf8828244684ab7b9be994b0d5d7, State: Initialized, Role: FOLLOWER
I20260924 11:47:21.517482 3526 tablet_replica.cc:330] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: stopping tablet replica
I20260924 11:47:21.517885 3526 raft_consensus.cc:2229] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:21.518347 3526 raft_consensus.cc:2258] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:21.518949 3528 tablet_bootstrap.cc:492] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: No bootstrap required, opened a new log
I20260924 11:47:21.518756 3529 consensus_queue.cc:260] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.519161 3528 ts_tablet_manager.cc:1397] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20260924 11:47:21.519953 3527 ts_tablet_manager.cc:1918] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:21.520159 3527 log.cc:1196] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/68d3b0b4cfd04f2f8851b8971fc3c31f
I20260924 11:47:21.520046 3528 raft_consensus.cc:348] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.520454 3528 raft_consensus.cc:374] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:21.520589 3528 raft_consensus.cc:729] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea86f9982ff4290a7310cae63e2b4f3, State: Initialized, Role: FOLLOWER
I20260924 11:47:21.520965 3527 ts_tablet_manager.cc:1939] T 68d3b0b4cfd04f2f8851b8971fc3c31f P aea86f9982ff4290a7310cae63e2b4f3: Deleting consensus metadata
I20260924 11:47:21.520889 3528 consensus_queue.cc:260] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.521315 3529 ts_tablet_manager.cc:1428] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260924 11:47:21.523239 2976 catalog_manager.cc:4841] TS aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325): tablet 68d3b0b4cfd04f2f8851b8971fc3c31f (table test-table [id=9383f971b3d74f489ec2e83c0d36a289]) successfully deleted
I20260924 11:47:21.523746 3528 ts_tablet_manager.cc:1428] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:21.526994 3526 ts_tablet_manager.cc:1905] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:21.528561 3525 ts_tablet_manager.cc:1918] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:21.528725 3525 log.cc:1196] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/68d3b0b4cfd04f2f8851b8971fc3c31f
I20260924 11:47:21.529270 3525 ts_tablet_manager.cc:1939] T 68d3b0b4cfd04f2f8851b8971fc3c31f P 81a8291168d1478690a617495105a2fb: Deleting consensus metadata
I20260924 11:47:21.530889 2974 catalog_manager.cc:4841] TS 81a8291168d1478690a617495105a2fb (127.1.210.1:41761): tablet 68d3b0b4cfd04f2f8851b8971fc3c31f (table test-table [id=9383f971b3d74f489ec2e83c0d36a289]) successfully deleted
I20260924 11:47:21.531783 3526 ts_tablet_manager.cc:1918] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:21.531941 3526 log.cc:1196] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/68d3b0b4cfd04f2f8851b8971fc3c31f
I20260924 11:47:21.532331 3526 ts_tablet_manager.cc:1939] T 68d3b0b4cfd04f2f8851b8971fc3c31f P a0dcbf8828244684ab7b9be994b0d5d7: Deleting consensus metadata
I20260924 11:47:21.533643 2973 catalog_manager.cc:4841] TS a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241): tablet 68d3b0b4cfd04f2f8851b8971fc3c31f (table test-table [id=9383f971b3d74f489ec2e83c0d36a289]) successfully deleted
I20260924 11:47:21.608033 3515 raft_consensus.cc:482] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:21.608256 3515 raft_consensus.cc:504] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.609899 3515 leader_election.cc:288] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241), aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:21.618911 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.610687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:41245 (local address 127.1.210.2:38241)
0924 11:47:21.610916 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.610924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.611663 (+ 739us) server_negotiation.cc:407] Connection header received
0924 11:47:21.611971 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.611988 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.612046 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.612143 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:21.612792 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.613614 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.616036 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.617312 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.617438 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.617533 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.618490 (+ 957us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.618636 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:47:21.619123 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.610466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:60671 (local address 127.1.210.3:38325)
0924 11:47:21.610801 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.610809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.611909 (+ 1100us) server_negotiation.cc:407] Connection header received
0924 11:47:21.612182 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.612189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.612253 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.612367 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:21.612947 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.614055 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.616324 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.617411 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.617539 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.617634 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.618658 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.618831 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260924 11:47:21.619213 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.610089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:38241 (local address 127.1.210.1:41245)
0924 11:47:21.611549 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.611575 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.611806 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.612173 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.612178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.612200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:21.612610 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.612625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.613736 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.613743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.615878 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.615891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.617525 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.617531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.618311 (+ 780us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.618352 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.618466 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.618643 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":37,"thread_start_us":1432,"threads_started":1}
I20260924 11:47:21.619889 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.610360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38325 (local address 127.1.210.1:60671)
0924 11:47:21.611815 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.611835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.612015 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.612383 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.612388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.612409 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:21.612773 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.612783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.614188 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.614194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.616183 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.616201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.618249 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.618256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.618468 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.618505 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.619358 (+ 853us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.619507 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"mutex_wait_us":58,"thread_start_us":806,"threads_started":1}
I20260924 11:47:21.619997 3290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1bc1469e844f02a228338d1acbfc4d" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" is_pre_election: true
I20260924 11:47:21.620275 3290 raft_consensus.cc:2454] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 0.
I20260924 11:47:21.620577 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1bc1469e844f02a228338d1acbfc4d" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea86f9982ff4290a7310cae63e2b4f3" is_pre_election: true
I20260924 11:47:21.620824 3425 raft_consensus.cc:2454] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 0.
I20260924 11:47:21.620791 3076 leader_election.cc:302] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a8291168d1478690a617495105a2fb, a0dcbf8828244684ab7b9be994b0d5d7; no voters:
I20260924 11:47:21.621260 3515 raft_consensus.cc:2781] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:21.621392 3515 raft_consensus.cc:482] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:21.621520 3515 raft_consensus.cc:3037] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.623513 3515 raft_consensus.cc:504] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.624055 3515 leader_election.cc:288] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 election: Requested vote from peers a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241), aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:21.624456 3290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1bc1469e844f02a228338d1acbfc4d" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dcbf8828244684ab7b9be994b0d5d7"
I20260924 11:47:21.624482 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1bc1469e844f02a228338d1acbfc4d" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea86f9982ff4290a7310cae63e2b4f3"
I20260924 11:47:21.624656 3425 raft_consensus.cc:3037] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.624666 3290 raft_consensus.cc:3037] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.627583 3290 raft_consensus.cc:2454] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 1.
I20260924 11:47:21.627924 3076 leader_election.cc:302] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a8291168d1478690a617495105a2fb, a0dcbf8828244684ab7b9be994b0d5d7; no voters:
I20260924 11:47:21.628206 3515 raft_consensus.cc:2781] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:21.628206 3425 raft_consensus.cc:2454] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 1.
I20260924 11:47:21.629302 3515 raft_consensus.cc:686] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 1 LEADER]: Becoming Leader. State: Replica: 81a8291168d1478690a617495105a2fb, State: Running, Role: LEADER
I20260924 11:47:21.629671 3515 consensus_queue.cc:237] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.633108 2988 catalog_manager.cc:5495] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a8291168d1478690a617495105a2fb (127.1.210.1). New cstate: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:47:21 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250439, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:21.673708 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.659316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42657 (local address 127.0.0.1:60868)
0924 11:47:21.660059 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.660078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.660234 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.660534 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.660539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.661413 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:21.661689 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.661700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.662765 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.662772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.663721 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.663732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.664440 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.664446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.664648 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.665334 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:21.665352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:21.667673 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:21.671050 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:21.671062 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:21.671070 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:21.671460 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:21.671805 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:21.671811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:21.671814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:21.672046 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:21.672590 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:21.672597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:21.672753 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.672921 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.673234 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":466,"threads_started":1}
I20260924 11:47:21.673727 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.659240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.1.210.62:42657)
0924 11:47:21.659900 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.659906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.660159 (+ 253us) server_negotiation.cc:407] Connection header received
0924 11:47:21.660345 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.660353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.660410 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.660515 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:21.661832 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.662649 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.663881 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.664270 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.664418 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.667870 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:21.667875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:21.667878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:21.667904 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:21.670905 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:21.671573 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:21.671581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:21.671585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:21.671701 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:21.672176 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:21.672180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:21.672184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:21.672448 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:21.672599 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.673003 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.673348 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":474,"thread_start_us":314,"threads_started":1}
I20260924 11:47:21.679176 2988 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:60868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:47:21.680169 2988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:21.687510 3126 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1e424cf52746b8803ba5379e613811 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1716951816d44b339ac43a1abfeb8e04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:47:21.687706 3403 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1e424cf52746b8803ba5379e613811 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1716951816d44b339ac43a1abfeb8e04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:47:21.687770 3265 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1e424cf52746b8803ba5379e613811 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1716951816d44b339ac43a1abfeb8e04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:47:21.688092 3126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1e424cf52746b8803ba5379e613811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:21.688190 3403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1e424cf52746b8803ba5379e613811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:21.688262 3265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1e424cf52746b8803ba5379e613811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:21.695238 3529 tablet_bootstrap.cc:492] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: Bootstrap starting.
I20260924 11:47:21.695763 3528 tablet_bootstrap.cc:492] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: Bootstrap starting.
I20260924 11:47:21.696866 3530 tablet_bootstrap.cc:492] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: Bootstrap starting.
I20260924 11:47:21.696942 3529 tablet_bootstrap.cc:654] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:21.697345 3528 tablet_bootstrap.cc:654] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:21.698823 3530 tablet_bootstrap.cc:654] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:21.702435 3529 tablet_bootstrap.cc:492] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: No bootstrap required, opened a new log
I20260924 11:47:21.702577 3529 ts_tablet_manager.cc:1397] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.003s
I20260924 11:47:21.703107 3530 tablet_bootstrap.cc:492] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: No bootstrap required, opened a new log
I20260924 11:47:21.703215 3530 ts_tablet_manager.cc:1397] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 11:47:21.703251 3529 raft_consensus.cc:348] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.703505 3529 raft_consensus.cc:374] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:21.703578 3529 raft_consensus.cc:729] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dcbf8828244684ab7b9be994b0d5d7, State: Initialized, Role: FOLLOWER
I20260924 11:47:21.703806 3529 consensus_queue.cc:260] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.703883 3530 raft_consensus.cc:348] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.704113 3530 raft_consensus.cc:374] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:21.704182 3530 raft_consensus.cc:729] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a8291168d1478690a617495105a2fb, State: Initialized, Role: FOLLOWER
I20260924 11:47:21.704309 3529 ts_tablet_manager.cc:1428] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:47:21.704254 3499 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.704530 3499 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.704984 3530 consensus_queue.cc:260] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.705408 3515 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.705674 3515 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.706449 3530 ts_tablet_manager.cc:1428] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:21.713852 3528 tablet_bootstrap.cc:492] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: No bootstrap required, opened a new log
I20260924 11:47:21.714036 3528 ts_tablet_manager.cc:1397] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20260924 11:47:21.714857 3528 raft_consensus.cc:348] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.715181 3528 raft_consensus.cc:374] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:21.715282 3528 raft_consensus.cc:729] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea86f9982ff4290a7310cae63e2b4f3, State: Initialized, Role: FOLLOWER
I20260924 11:47:21.715541 3528 consensus_queue.cc:260] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.716159 3528 ts_tablet_manager.cc:1428] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:47:21.716071 3520 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.716568 3520 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.726660 3515 raft_consensus.cc:482] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:21.726894 3515 raft_consensus.cc:504] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.727561 3515 leader_election.cc:288] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241), aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:21.728051 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1e424cf52746b8803ba5379e613811" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea86f9982ff4290a7310cae63e2b4f3" is_pre_election: true
I20260924 11:47:21.728042 3290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1e424cf52746b8803ba5379e613811" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" is_pre_election: true
I20260924 11:47:21.728267 3290 raft_consensus.cc:2454] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 0.
I20260924 11:47:21.728302 3425 raft_consensus.cc:2454] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 0.
I20260924 11:47:21.728718 3076 leader_election.cc:302] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a8291168d1478690a617495105a2fb, a0dcbf8828244684ab7b9be994b0d5d7; no voters:
I20260924 11:47:21.729018 3515 raft_consensus.cc:2781] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:21.729140 3515 raft_consensus.cc:482] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:21.729247 3515 raft_consensus.cc:3037] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.732643 3515 raft_consensus.cc:504] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.733331 3515 leader_election.cc:288] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 election: Requested vote from peers a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241), aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325)
I20260924 11:47:21.733727 3425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1e424cf52746b8803ba5379e613811" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea86f9982ff4290a7310cae63e2b4f3"
I20260924 11:47:21.733773 3290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1e424cf52746b8803ba5379e613811" candidate_uuid: "81a8291168d1478690a617495105a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dcbf8828244684ab7b9be994b0d5d7"
I20260924 11:47:21.733949 3425 raft_consensus.cc:3037] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.733987 3290 raft_consensus.cc:3037] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:21.736171 3425 raft_consensus.cc:2454] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 1.
I20260924 11:47:21.736639 3079 leader_election.cc:302] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a8291168d1478690a617495105a2fb, aea86f9982ff4290a7310cae63e2b4f3; no voters:
I20260924 11:47:21.736873 3290 raft_consensus.cc:2454] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a8291168d1478690a617495105a2fb in term 1.
I20260924 11:47:21.736912 3515 raft_consensus.cc:2781] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:21.737507 3515 raft_consensus.cc:686] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [term 1 LEADER]: Becoming Leader. State: Replica: 81a8291168d1478690a617495105a2fb, State: Running, Role: LEADER
I20260924 11:47:21.737749 3515 consensus_queue.cc:237] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } }
I20260924 11:47:21.739408 3536 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: TxnStatusTablet state changed. Reason: New leader 81a8291168d1478690a617495105a2fb. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.739797 3536 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:47:21.741272 3549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:47:21.741626 3549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:47:21.744793 2988 catalog_manager.cc:5495] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a8291168d1478690a617495105a2fb (127.1.210.1). New cstate: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:21.797474 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.788538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:41761 (local address 127.0.0.1:47610)
0924 11:47:21.789266 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.789290 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.789458 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.790409 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.790415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.790430 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:47:21.790652 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.790664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.792414 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.792419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.794258 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.794274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.795360 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.795366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.795573 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.795646 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:47:21.796829 (+ 1183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:47:21.796867 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.796966 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.797149 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481}
I20260924 11:47:21.798069 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.789791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.1.210.1:41761)
0924 11:47:21.789970 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.789976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.789997 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:47:21.790085 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.790091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.790183 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.790339 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:21.790853 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.792229 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.794456 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.795167 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.795332 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.795870 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:21.796644 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:21.796809 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.797702 (+ 893us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.797797 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:47:21.802553 3290 raft_consensus.cc:1261] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Refusing update from remote peer 81a8291168d1478690a617495105a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:21.802614 3425 raft_consensus.cc:1261] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Refusing update from remote peer 81a8291168d1478690a617495105a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:21.803344 3536 consensus_queue.cc:1035] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:21.803649 3515 consensus_queue.cc:1035] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:21.817821 3520 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: TxnStatusTablet state changed. Reason: New leader 81a8291168d1478690a617495105a2fb. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.818142 3520 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.818059 3499 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: TxnStatusTablet state changed. Reason: New leader 81a8291168d1478690a617495105a2fb. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.818368 3499 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.821542 3536 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.821821 3536 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:47:21.822553 3536 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.822778 3536 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P 81a8291168d1478690a617495105a2fb: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:47:21.823967 3517 mvcc.cc:205] Tried to move back new op lower bound from 7332865809615933440 to 7332865809359884288. Current Snapshot: MvccSnapshot[applied={T|T < 7332865809615933440}]
I20260924 11:47:21.824903 3499 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.825181 3499 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P a0dcbf8828244684ab7b9be994b0d5d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.829788 3520 tablet_replica.cc:439] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81a8291168d1478690a617495105a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a8291168d1478690a617495105a2fb" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 41761 } } peers { permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 } } peers { permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 } } }
I20260924 11:47:21.830086 3520 tablet_replica.cc:442] T 0e1e424cf52746b8803ba5379e613811 P aea86f9982ff4290a7310cae63e2b4f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:21.839090 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.832400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.1.210.1:41761)
0924 11:47:21.832559 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.832565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.832582 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:47:21.832640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.832645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.832721 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.832822 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:21.833415 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.834615 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.836547 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.837251 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.837381 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.837952 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:21.838100 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:21.838184 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.838736 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.838810 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:47:21.860282 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.851926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:50283 (local address 127.1.210.1:41761)
0924 11:47:21.852106 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:21.852113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:21.852486 (+ 373us) server_negotiation.cc:407] Connection header received
0924 11:47:21.852716 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:21.852721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:21.852795 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:21.852899 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:21.853749 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.854885 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.857932 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.858722 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.858842 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.859405 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:21.859544 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:21.859630 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:21.859853 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 11:47:21.859928 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:47:21.860900 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:21.851547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:41761 (local address 127.1.210.1:50283)
0924 11:47:21.852362 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:21.852387 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:21.852569 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:21.852986 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:21.852993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:21.853009 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:47:21.853568 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.853584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.855479 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.855486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.857764 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:21.857784 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:21.858897 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:21.858905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:21.859105 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:21.859196 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:47:21.859672 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:47:21.859693 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:47:21.859807 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 11:47:21.859980 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":572,"threads_started":1}
I20260924 11:47:21.875624 3425 raft_consensus.cc:1261] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Refusing update from remote peer 81a8291168d1478690a617495105a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:21.876263 3536 consensus_queue.cc:1035] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea86f9982ff4290a7310cae63e2b4f3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:21.889959 3289 raft_consensus.cc:1261] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Refusing update from remote peer 81a8291168d1478690a617495105a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:21.890851 3515 consensus_queue.cc:1035] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0dcbf8828244684ab7b9be994b0d5d7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:21.892693 3514 mvcc.cc:205] Tried to move back new op lower bound from 7332865809911693312 to 7332865808918659072. Current Snapshot: MvccSnapshot[applied={T|T < 7332865809911693312}]
I20260924 11:47:21.974140 2988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:21.974400 2988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:21.977893 2988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db7d3aad44f04930a87125b1ee3b0020: Sending DeleteTablet for 3 replicas of tablet 3c1bc1469e844f02a228338d1acbfc4d
I20260924 11:47:21.979027 3403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1bc1469e844f02a228338d1acbfc4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:21 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:52068
I20260924 11:47:21.979413 3265 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1bc1469e844f02a228338d1acbfc4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:21 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:35930
I20260924 11:47:21.979730 3526 tablet_replica.cc:330] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: stopping tablet replica
I20260924 11:47:21.979775 3527 tablet_replica.cc:330] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: stopping tablet replica
I20260924 11:47:21.979949 3526 raft_consensus.cc:2229] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:21.980019 3527 raft_consensus.cc:2229] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:21.980139 3526 raft_consensus.cc:2258] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:21.980255 3527 raft_consensus.cc:2258] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:21.981513 3526 ts_tablet_manager.cc:1905] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:21.981843 3527 ts_tablet_manager.cc:1905] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:21.988432 3526 ts_tablet_manager.cc:1918] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:21.988559 3526 log.cc:1196] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/3c1bc1469e844f02a228338d1acbfc4d
I20260924 11:47:21.988934 3526 ts_tablet_manager.cc:1939] T 3c1bc1469e844f02a228338d1acbfc4d P a0dcbf8828244684ab7b9be994b0d5d7: Deleting consensus metadata
I20260924 11:47:21.990113 2973 catalog_manager.cc:4841] TS a0dcbf8828244684ab7b9be994b0d5d7 (127.1.210.2:38241): tablet 3c1bc1469e844f02a228338d1acbfc4d (table test-table [id=3a5b544fd4c140d297decbd40be40caf]) successfully deleted
I20260924 11:47:21.990263 3126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1bc1469e844f02a228338d1acbfc4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:21 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:47608
I20260924 11:47:21.991441 3525 tablet_replica.cc:330] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: stopping tablet replica
I20260924 11:47:21.991693 3525 raft_consensus.cc:2229] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:21.992249 3525 raft_consensus.cc:2258] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:21.994091 3525 ts_tablet_manager.cc:1905] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:22.002461 3527 ts_tablet_manager.cc:1918] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:22.002768 3525 ts_tablet_manager.cc:1918] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:22.002969 3525 log.cc:1196] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/3c1bc1469e844f02a228338d1acbfc4d
I20260924 11:47:22.003489 3525 ts_tablet_manager.cc:1939] T 3c1bc1469e844f02a228338d1acbfc4d P 81a8291168d1478690a617495105a2fb: Deleting consensus metadata
I20260924 11:47:22.004706 3527 log.cc:1196] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/3c1bc1469e844f02a228338d1acbfc4d
I20260924 11:47:22.004971 2974 catalog_manager.cc:4841] TS 81a8291168d1478690a617495105a2fb (127.1.210.1:41761): tablet 3c1bc1469e844f02a228338d1acbfc4d (table test-table [id=3a5b544fd4c140d297decbd40be40caf]) successfully deleted
I20260924 11:47:22.005422 3527 ts_tablet_manager.cc:1939] T 3c1bc1469e844f02a228338d1acbfc4d P aea86f9982ff4290a7310cae63e2b4f3: Deleting consensus metadata
I20260924 11:47:22.006546 2976 catalog_manager.cc:4841] TS aea86f9982ff4290a7310cae63e2b4f3 (127.1.210.3:38325): tablet 3c1bc1469e844f02a228338d1acbfc4d (table test-table [id=3a5b544fd4c140d297decbd40be40caf]) successfully deleted
Sep 24 11:47:22 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:22.028754 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:22.003390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.1.210.1:41761)
0924 11:47:22.003567 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:22.003574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:22.006053 (+ 2479us) server_negotiation.cc:407] Connection header received
0924 11:47:22.006427 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:22.006433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:22.006587 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:22.006705 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:22.009323 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.010592 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.012734 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.013567 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.013719 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:22.020597 (+ 6878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:22.020614 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:22.020639 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:22.020692 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:22.025357 (+ 4665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.026293 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.026303 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.026307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.026450 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.026910 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.026915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.026919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.027381 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:22.027677 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:22.028098 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 11:47:22.028330 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 11:47:22.029590 3191 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47628
I20260924 11:47:22.056913 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:22.039947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.1.210.62:42657)
0924 11:47:22.040180 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:22.040188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:22.040479 (+ 291us) server_negotiation.cc:407] Connection header received
0924 11:47:22.040710 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:22.040717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:22.040809 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:22.040954 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:22.043105 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.044415 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.046252 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.046986 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.047134 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:22.050063 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:22.050072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:22.050078 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:22.050124 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:22.053774 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.054961 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.054966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.054968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.055101 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.055464 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.055468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.055470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.055761 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:22.055964 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:22.056371 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 11:47:22.056534 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 11:47:22.057585 2988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870
I20260924 11:47:22.059648 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3060
I20260924 11:47:22.074472 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:22.060887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.1.210.1:41761)
0924 11:47:22.061117 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:22.061126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:22.061392 (+ 266us) server_negotiation.cc:407] Connection header received
0924 11:47:22.061558 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:22.061564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:22.061658 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:22.061767 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:22.063001 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.064035 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.065182 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.065871 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.065986 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:22.067944 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:22.067950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:22.067954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:22.067987 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:22.071862 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.072448 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.072453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.072457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.072608 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.072924 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.072929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.072933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.073332 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:22.073602 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:22.073888 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 11:47:22.074099 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 11:47:22.075131 3191 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47642
W20260924 11:47:22.075644 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3060: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:22.075815 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3060
I20260924 11:47:22.094770 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3200
Sep 24 11:47:22 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:22.109856 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:22.095531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.1.210.2:38241)
0924 11:47:22.095697 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:22.095707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:22.095723 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:47:22.095897 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:22.095901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:22.095971 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:22.096052 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:22.097542 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.098352 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.099694 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.100080 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.100172 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:22.104578 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:22.104588 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:22.104602 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:22.104631 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:22.107299 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.107913 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.107922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.107925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.108075 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.108416 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.108420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.108422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.108768 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:22.109008 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:22.109367 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 11:47:22.109538 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 11:47:22.110455 3330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35946
W20260924 11:47:22.111012 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:22.111145 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3200
I20260924 11:47:22.127964 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3339
Sep 24 11:47:22 dist-test-slave-dd32 krb5kdc[2944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:22.142068 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:22.128633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.1.210.3:38325)
0924 11:47:22.129164 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:22.129169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:22.129186 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:47:22.129235 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:22.129238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:22.129299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:22.129387 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:22.130478 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.131218 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.132338 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:22.132736 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:22.132820 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:22.136616 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:22.136630 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:22.136644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:22.136672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:22.139474 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.140116 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.140122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.140124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.140274 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:22.140663 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:22.140667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:22.140685 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:22.141051 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:22.141330 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:22.141527 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 11:47:22.141683 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
W20260924 11:47:22.142688 3469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52086
W20260924 11:47:22.143287 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3339: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:22.143438 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3339
I20260924 11:47:22.164109 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 2957
I20260924 11:47:22.165330 3029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:22.276973 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 2957
2026-09-24T11:47:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4795 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:22 dist-test-slave-dd32 krb5kdc[3570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:22 dist-test-slave-dd32 krb5kdc[3570](info): set up 2 sockets
Sep 24 11:47:22 dist-test-slave-dd32 krb5kdc[3570](info): commencing operation
krb5kdc: starting...
W20260924 11:47:23.365617 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:23 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:23.544493 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:33025
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:23.852660 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:23.852931 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:23.853152 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:23.860234 3583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:23.860325 3583 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:47:23.860397 3583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:23.860493 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:23.860580 3583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:23.860685 3583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:23.870189 3583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:33025
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:23.871369 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:23.872651 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:23.884100 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:23.884073 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:23.884063 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:23.884909 3583 server_base.cc:1034] running on GCE node
I20260924 11:47:23.885577 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:23.886744 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:23.887964 3583 hybrid_clock.cc:648] HybridClock initialized: now 1790250443887887 us; error 46 us; skew 500 ppm
I20260924 11:47:23.888280 3583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:23 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250443, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:23.891425 3583 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:23.892728 3583 webserver.cc:460] Webserver started at http://127.1.210.62:34837/ using document root <none> and password file <none>
I20260924 11:47:23.893460 3583 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:23.893587 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:23.893893 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:23.896649 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "e618ec21c32e41cb98a27b21adbb34b9"
format_stamp: "Formatted at 2026-09-24 11:47:23 on dist-test-slave-dd32"
server_key: "493507d1933e23292a80a904a20a5ed0"
server_key_iv: "758146f5cb64f8ad5118d6b9871a67b6"
server_key_version: "encryptionkey@0"
I20260924 11:47:23.897445 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "e618ec21c32e41cb98a27b21adbb34b9"
format_stamp: "Formatted at 2026-09-24 11:47:23 on dist-test-slave-dd32"
server_key: "493507d1933e23292a80a904a20a5ed0"
server_key_iv: "758146f5cb64f8ad5118d6b9871a67b6"
server_key_version: "encryptionkey@0"
I20260924 11:47:23.902690 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:47:23.905850 3598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:23.907054 3583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:47:23.907227 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "e618ec21c32e41cb98a27b21adbb34b9"
format_stamp: "Formatted at 2026-09-24 11:47:23 on dist-test-slave-dd32"
server_key: "493507d1933e23292a80a904a20a5ed0"
server_key_iv: "758146f5cb64f8ad5118d6b9871a67b6"
server_key_version: "encryptionkey@0"
I20260924 11:47:23.907429 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:23.927600 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:23.931938 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:23.932235 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:23.952474 3583 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:33025
I20260924 11:47:23.952492 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:33025 every 8 connection(s)
I20260924 11:47:23.953758 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:23.957293 3660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:23.959942 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3583
I20260924 11:47:23.960148 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:23.960762 1864 external_mini_cluster.cc:1428] Setting key 631f2dfbb914090300aa832e882074fa
I20260924 11:47:23.969077 3660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Bootstrap starting.
I20260924 11:47:23.972611 3660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:23.973912 3660 log.cc:824] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:47:23 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:23.976881 3660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: No bootstrap required, opened a new log
I20260924 11:47:23.983772 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:23.962772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.1.210.62:33025)
0924 11:47:23.963529 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:23.963540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:23.963577 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:47:23.965207 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:23.965230 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:23.965587 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:23.965895 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:23.967578 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:23.968789 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:23.970498 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:23.971150 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:23.971281 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:23.975685 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:23.975695 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:23.975712 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:23.975753 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:23.979558 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:23.980223 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:23.980231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:23.980240 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:23.980443 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:23.980761 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:23.980766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:23.980769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:23.981472 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:23.981753 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:23.982049 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:47:23.982473 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260924 11:47:23.987033 3660 raft_consensus.cc:348] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618ec21c32e41cb98a27b21adbb34b9" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33025 } }
I20260924 11:47:23.987339 3660 raft_consensus.cc:374] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:23.987411 3660 raft_consensus.cc:729] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e618ec21c32e41cb98a27b21adbb34b9, State: Initialized, Role: FOLLOWER
I20260924 11:47:23.987843 3660 consensus_queue.cc:260] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618ec21c32e41cb98a27b21adbb34b9" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33025 } }
I20260924 11:47:23.988035 3660 raft_consensus.cc:388] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:23.988127 3660 raft_consensus.cc:482] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:23.988233 3660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:23.990238 3660 raft_consensus.cc:504] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618ec21c32e41cb98a27b21adbb34b9" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33025 } }
I20260924 11:47:23.990758 3660 leader_election.cc:302] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e618ec21c32e41cb98a27b21adbb34b9; no voters:
I20260924 11:47:23.991330 3660 leader_election.cc:288] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:23.991434 3665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:23.992214 3665 raft_consensus.cc:686] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [term 1 LEADER]: Becoming Leader. State: Replica: e618ec21c32e41cb98a27b21adbb34b9, State: Running, Role: LEADER
I20260924 11:47:23.992748 3665 consensus_queue.cc:237] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618ec21c32e41cb98a27b21adbb34b9" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33025 } }
I20260924 11:47:23.993366 3660 sys_catalog.cc:564] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:23.995473 3667 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e618ec21c32e41cb98a27b21adbb34b9. Latest consensus state: current_term: 1 leader_uuid: "e618ec21c32e41cb98a27b21adbb34b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618ec21c32e41cb98a27b21adbb34b9" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33025 } } }
I20260924 11:47:23.995621 3666 sys_catalog.cc:455] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e618ec21c32e41cb98a27b21adbb34b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e618ec21c32e41cb98a27b21adbb34b9" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33025 } } }
I20260924 11:47:23.995872 3666 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:23.995872 3667 sys_catalog.cc:458] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:23.996758 3673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:24.001478 3673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:24.009398 3673 catalog_manager.cc:1347] Generated new cluster ID: 8e3e37d7960348e99a2c98200246489f
I20260924 11:47:24.009509 3673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:24.034582 3673 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:24.036345 3673 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:24.044050 3673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Generated new TSK 0
I20260924 11:47:24.045090 3673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:24.092274 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33025
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:24.406373 3686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:24.406634 3686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:24.406730 3686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:24.413960 3686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:24.414065 3686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:24.414146 3686 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:47:24.414196 3686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:47:24.414268 3686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:24.423800 3686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:24.425009 3686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:24.426291 3686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:24.438966 3694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:24.438930 3692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:24.439016 3691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:24.439910 3686 server_base.cc:1034] running on GCE node
I20260924 11:47:24.440445 3686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:24.441386 3686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:24.442577 3686 hybrid_clock.cc:648] HybridClock initialized: now 1790250444442520 us; error 51 us; skew 500 ppm
I20260924 11:47:24.442828 3686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:24 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250444, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:24.451588 3686 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:24.453107 3686 webserver.cc:460] Webserver started at http://127.1.210.1:35881/ using document root <none> and password file <none>
I20260924 11:47:24.453811 3686 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:24.453959 3686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:24.454272 3686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:24.457127 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "f8be1202d2fc4120a41b27fc188d96f1"
format_stamp: "Formatted at 2026-09-24 11:47:24 on dist-test-slave-dd32"
server_key: "bee1910368963aa032b88450ad1f5c30"
server_key_iv: "764d1b12b03f424705b1a5614d7bf92a"
server_key_version: "encryptionkey@0"
I20260924 11:47:24.457881 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "f8be1202d2fc4120a41b27fc188d96f1"
format_stamp: "Formatted at 2026-09-24 11:47:24 on dist-test-slave-dd32"
server_key: "bee1910368963aa032b88450ad1f5c30"
server_key_iv: "764d1b12b03f424705b1a5614d7bf92a"
server_key_version: "encryptionkey@0"
I20260924 11:47:24.462767 3686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 11:47:24.466045 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:24.467211 3686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:47:24.467410 3686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "f8be1202d2fc4120a41b27fc188d96f1"
format_stamp: "Formatted at 2026-09-24 11:47:24 on dist-test-slave-dd32"
server_key: "bee1910368963aa032b88450ad1f5c30"
server_key_iv: "764d1b12b03f424705b1a5614d7bf92a"
server_key_version: "encryptionkey@0"
I20260924 11:47:24.467597 3686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:24.482815 3686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:24.486595 3686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:24.486866 3686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:24.489471 3686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:24.489599 3686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:24.489784 3686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:24.489893 3686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:47:24 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250444, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:24.513046 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:24.490410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.1.210.1:47375)
0924 11:47:24.491289 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:24.491339 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:24.493430 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:24.494209 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:24.494222 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:24.495073 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:24.495870 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:24.495889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.497604 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.497610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:24.498956 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:24.498966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.500093 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.500101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:24.500317 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:24.501022 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:24.501042 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:24.503947 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:24.507977 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:24.507989 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:24.508005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:24.508560 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:24.508995 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:24.509005 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:24.509013 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:24.509253 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:24.509917 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:24.509924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:24.510424 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 11:47:24.510801 (+ 377us) client_negotiation.cc:240] Negotiation successful
0924 11:47:24.511362 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":366,"threads_started":1}
I20260924 11:47:24.514084 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:24.489839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:47375 (local address 127.1.210.62:33025)
0924 11:47:24.490322 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:24.490328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:24.491498 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 11:47:24.493712 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:24.493720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:24.493833 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:24.493968 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:24.496073 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.497399 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.499090 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.499848 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.499983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:24.504182 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:24.504189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:24.504194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:24.504234 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:24.507844 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:24.508693 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:24.508700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:24.508706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:24.508871 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:24.509371 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:24.509378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:24.509384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:24.509772 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:24.509934 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:24.513655 (+ 3721us) server_negotiation.cc:299] Negotiation successful
0924 11:47:24.513861 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260924 11:47:24.538482 3686 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:39787
I20260924 11:47:24.538553 3821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:39787 every 8 connection(s)
I20260924 11:47:24.540109 3686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:24.540359 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:24.524784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:45879 (local address 127.1.210.62:33025)
0924 11:47:24.524986 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:24.524994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:24.526148 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 11:47:24.526358 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:24.526365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:24.526462 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:24.526577 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:24.527566 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.528797 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.530681 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.531475 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.531611 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:24.534116 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:24.534126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:24.534132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:24.534175 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:24.537550 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:24.538538 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:24.538545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:24.538550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:24.538706 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:24.539091 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:24.539094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:24.539096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:24.539343 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:24.539475 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:24.539912 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 11:47:24.540066 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:47:24.540607 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:24.524912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.1.210.1:45879)
0924 11:47:24.526000 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:24.526032 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:24.526212 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:24.526624 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:24.526632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:24.527124 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:24.527416 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:24.527425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.528935 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.528947 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:24.530461 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:24.530484 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:24.531887 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:24.531896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:24.532149 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:24.533000 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:24.533028 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:24.533927 (+ 899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:24.537698 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:24.537706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:24.537713 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:24.538373 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:24.538818 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:24.538823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:24.538827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:24.538970 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:24.539466 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:24.539471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:24.539682 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 11:47:24.539826 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:47:24.540163 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":717,"threads_started":1}
I20260924 11:47:24.542438 3822 heartbeater.cc:347] Connected to a master server at 127.1.210.62:33025
I20260924 11:47:24.542676 3822 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:24.543531 3822 heartbeater.cc:511] Master 127.1.210.62:33025 requested a full tablet report, sending...
I20260924 11:47:24.545662 3614 ts_manager.cc:194] Registered new tserver with Master: f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787)
I20260924 11:47:24.546989 3614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:47375
I20260924 11:47:24.549474 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3686
I20260924 11:47:24.549672 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:24.550264 1864 external_mini_cluster.cc:1428] Setting key 94cbbb2942bc108a1892ae7a8735761a
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:24.588239 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33025
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:24.905927 3826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:24.906198 3826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:24.906306 3826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:24.913573 3826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:24.913673 3826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:24.913753 3826 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:47:24.913802 3826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:47:24.913869 3826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:24.923673 3826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:24.924849 3826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:24.926123 3826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:24.941567 3832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:24.941661 3834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:24.941612 3831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:24.942608 3826 server_base.cc:1034] running on GCE node
I20260924 11:47:24.943257 3826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:24.944198 3826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:24.945422 3826 hybrid_clock.cc:648] HybridClock initialized: now 1790250444945359 us; error 36 us; skew 500 ppm
I20260924 11:47:24.945715 3826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:24 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250444, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:24.948623 3826 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:24.950083 3826 webserver.cc:460] Webserver started at http://127.1.210.2:40491/ using document root <none> and password file <none>
I20260924 11:47:24.950712 3826 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:24.950809 3826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:24.951053 3826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:24.953966 3826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "7ffd872059bf4404bc6927f8d4790f07"
format_stamp: "Formatted at 2026-09-24 11:47:24 on dist-test-slave-dd32"
server_key: "e72295b247082ad498ea0462c96367d7"
server_key_iv: "4322f083f2b2e39e4184b6dbbbfb1959"
server_key_version: "encryptionkey@0"
I20260924 11:47:24.954681 3826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "7ffd872059bf4404bc6927f8d4790f07"
format_stamp: "Formatted at 2026-09-24 11:47:24 on dist-test-slave-dd32"
server_key: "e72295b247082ad498ea0462c96367d7"
server_key_iv: "4322f083f2b2e39e4184b6dbbbfb1959"
server_key_version: "encryptionkey@0"
I20260924 11:47:24.959625 3826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:47:24.963006 3841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:24.964270 3826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 11:47:24.964440 3826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "7ffd872059bf4404bc6927f8d4790f07"
format_stamp: "Formatted at 2026-09-24 11:47:24 on dist-test-slave-dd32"
server_key: "e72295b247082ad498ea0462c96367d7"
server_key_iv: "4322f083f2b2e39e4184b6dbbbfb1959"
server_key_version: "encryptionkey@0"
I20260924 11:47:24.964630 3826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:24.998862 3826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:25.002635 3826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:25.002959 3826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:25.005591 3826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:25.005708 3826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:25.005817 3826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:25.005900 3826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:47:25 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250444, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:25.032399 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.006974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.1.210.2:57859)
0924 11:47:25.007749 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.007786 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.009681 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.010496 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.010510 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.011451 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:25.012407 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.012438 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.014367 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.014374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.016696 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.016714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.017677 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.017689 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.017993 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.018869 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:25.018924 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:25.022797 (+ 3873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:25.026939 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.026949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.026978 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.027637 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.028173 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.028178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.028181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.028470 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.029274 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:25.029282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:25.029804 (+ 522us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.030140 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.030789 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":361,"threads_started":1}
I20260924 11:47:25.033231 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.006224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57859 (local address 127.1.210.62:33025)
0924 11:47:25.006424 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.006432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.007899 (+ 1467us) server_negotiation.cc:407] Connection header received
0924 11:47:25.009997 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.010003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.010128 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.010240 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:25.012635 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.014153 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.016920 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.017429 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.017573 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.023088 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:25.023097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:25.023103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:25.023150 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:25.026765 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.027833 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.027838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.027840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.028026 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.028618 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.028623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.028625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.029028 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:25.029193 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.032499 (+ 3306us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.032812 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:47:25.049790 3826 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:40567
I20260924 11:47:25.049829 3960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:40567 every 8 connection(s)
I20260924 11:47:25.051004 3826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:25.052968 3961 heartbeater.cc:347] Connected to a master server at 127.1.210.62:33025
I20260924 11:47:25.053231 3961 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:25.054117 3961 heartbeater.cc:511] Master 127.1.210.62:33025 requested a full tablet report, sending...
I20260924 11:47:25.055131 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.040041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:45591 (local address 127.1.210.62:33025)
0924 11:47:25.040197 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.040201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.040844 (+ 643us) server_negotiation.cc:407] Connection header received
0924 11:47:25.041131 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.041139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.041224 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.041300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:25.042989 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.044044 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.045699 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.046266 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.046407 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.048851 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:25.048857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:25.048861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:25.048894 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:25.051973 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.052719 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.052726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.052731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.052880 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.053306 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.053312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.053318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.053793 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:25.053951 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.054535 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.054791 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:47:25.055148 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3826
I20260924 11:47:25.055212 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.040120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.1.210.2:45591)
0924 11:47:25.040747 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.040761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.040934 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.041540 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.041543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.042207 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:25.042753 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.042768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.044178 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.044183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.045568 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.045579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.046487 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.046499 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.046752 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.047515 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:25.047541 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:25.048642 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:25.052126 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.052130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.052134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.052592 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.052986 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.052991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.052994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.053179 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.053949 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:25.053953 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:25.054116 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.054442 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.054778 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":384,"threads_started":1}
I20260924 11:47:25.055333 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:25.055843 1864 external_mini_cluster.cc:1428] Setting key cd08bf986d2200feb2c02e48e3494dfd
I20260924 11:47:25.056999 3614 ts_manager.cc:194] Registered new tserver with Master: 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567)
I20260924 11:47:25.058429 3614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:57859
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:25.094853 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33025
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:25.414163 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:25.414420 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:25.414517 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:25.421569 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:25.421675 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:25.421764 3965 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:47:25.421818 3965 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:47:25.421886 3965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:25.431562 3965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:25.432760 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:25.434038 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:25.446986 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:25.447019 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:25.447515 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:25.448048 3965 server_base.cc:1034] running on GCE node
I20260924 11:47:25.448557 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:25.449483 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:25.450670 3965 hybrid_clock.cc:648] HybridClock initialized: now 1790250445450631 us; error 28 us; skew 500 ppm
I20260924 11:47:25.450971 3965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:25 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250445, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:25.453869 3965 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:25.455230 3965 webserver.cc:460] Webserver started at http://127.1.210.3:44969/ using document root <none> and password file <none>
I20260924 11:47:25.455866 3965 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:25.456012 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:25.456318 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:25.459067 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
format_stamp: "Formatted at 2026-09-24 11:47:25 on dist-test-slave-dd32"
server_key: "45a155fdac526789adc382dfe5b2e8fe"
server_key_iv: "f133a57b06cf6d0b058877caa5f006bb"
server_key_version: "encryptionkey@0"
I20260924 11:47:25.459784 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
format_stamp: "Formatted at 2026-09-24 11:47:25 on dist-test-slave-dd32"
server_key: "45a155fdac526789adc382dfe5b2e8fe"
server_key_iv: "f133a57b06cf6d0b058877caa5f006bb"
server_key_version: "encryptionkey@0"
I20260924 11:47:25.464884 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:47:25.468832 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:25.470224 3965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:25.470409 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
format_stamp: "Formatted at 2026-09-24 11:47:25 on dist-test-slave-dd32"
server_key: "45a155fdac526789adc382dfe5b2e8fe"
server_key_iv: "f133a57b06cf6d0b058877caa5f006bb"
server_key_version: "encryptionkey@0"
I20260924 11:47:25.470649 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:25.497731 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:25.501832 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:25.502144 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:25.504756 3965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:25.504917 3965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:25.505056 3965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:25.505187 3965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 11:47:25 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250445, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:25.529780 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.505632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:52317 (local address 127.1.210.62:33025)
0924 11:47:25.505823 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.505831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.507326 (+ 1495us) server_negotiation.cc:407] Connection header received
0924 11:47:25.509830 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.509836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.509923 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.510038 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:25.512447 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.513584 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.515993 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.516758 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.516897 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.522340 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:25.522347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:25.522352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:25.522393 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:25.526230 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.527253 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.527262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.527268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.527444 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.528004 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.528011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.528014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.528283 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:25.528462 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.529173 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.529394 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:47:25.531253 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.506258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.1.210.3:52317)
0924 11:47:25.507136 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.507184 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.509463 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.510296 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.510312 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.511232 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:25.512252 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.512281 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.513724 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.513730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.515787 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.515801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.517262 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.517272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.517640 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.518555 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:25.518598 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:25.522096 (+ 3498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:25.526379 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.526388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.526414 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.527080 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.527602 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.527613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.527617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.527872 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.528421 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:25.528428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:25.528771 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.529046 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.529620 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":390,"threads_started":1}
I20260924 11:47:25.549576 3822 heartbeater.cc:503] Master 127.1.210.62:33025 was elected leader, sending a full tablet report...
I20260924 11:47:25.560837 3965 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:36253
I20260924 11:47:25.560837 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:36253 every 8 connection(s)
I20260924 11:47:25.562449 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:25.564558 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.548652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:37049 (local address 127.1.210.62:33025)
0924 11:47:25.548873 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.548883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.549552 (+ 669us) server_negotiation.cc:407] Connection header received
0924 11:47:25.549846 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.549856 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.549959 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.550059 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:25.551745 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.552525 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.554274 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.555060 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.555210 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.557864 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:25.557874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:25.557882 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:25.557927 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:25.561671 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.562286 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.562291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.562294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.562453 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.562823 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.562828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.562834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.563209 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:25.563391 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.563991 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.564202 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:47:25.564541 4100 heartbeater.cc:347] Connected to a master server at 127.1.210.62:33025
I20260924 11:47:25.564611 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.548624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.1.210.3:37049)
0924 11:47:25.549430 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.549454 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.549649 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.550315 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.550323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.551102 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:25.551577 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.551593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.552662 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.552668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.554115 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.554132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.555296 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.555302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.555427 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.556096 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:25.556125 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:25.557606 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:25.561785 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.561788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.561791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.562155 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.562569 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.562573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.562576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.562705 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.563461 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:25.563466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:25.563705 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.563860 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.564161 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":397,"threads_started":1}
I20260924 11:47:25.564788 4100 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:25.565536 4100 heartbeater.cc:511] Master 127.1.210.62:33025 requested a full tablet report, sending...
I20260924 11:47:25.566846 3614 ts_manager.cc:194] Registered new tserver with Master: ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:25.567875 3614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:52317
I20260924 11:47:25.572728 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 3965
I20260924 11:47:25.572860 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:25.573297 1864 external_mini_cluster.cc:1428] Setting key 6f8b7fd786784da387e9a8f5cf98c2d4
I20260924 11:47:25.576085 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:47:25 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:47:25 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:25.620641 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.607979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.1.210.62:33025)
0924 11:47:25.608199 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.608206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.608347 (+ 141us) server_negotiation.cc:407] Connection header received
0924 11:47:25.608494 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.608498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.608560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.608644 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:25.610045 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.610818 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.612078 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.612480 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.612573 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.616201 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:25.616206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:25.616209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:25.616237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:25.618617 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.619151 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.619162 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.619166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.619274 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.619572 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.619575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.619577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.619812 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:25.619984 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.620182 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.620328 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:47:25.624571 3614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:25.631421 3614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:25.653800 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.642901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:36253 (local address 127.0.0.1:44652)
0924 11:47:25.643775 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.643813 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.643981 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.646804 (+ 2823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.646812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.646849 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:25.647286 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.647298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.649520 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.649526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.651165 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.651176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.652879 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.652887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.653090 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.653143 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.653275 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.653461 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":3681,"threads_started":1}
I20260924 11:47:25.653816 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.643062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.1.210.3:36253)
0924 11:47:25.646183 (+ 3121us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.646194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.646228 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:47:25.646323 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.646329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.646577 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.647524 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:25.647828 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.649375 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.651319 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.652700 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.652828 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.653091 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.653330 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.653483 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2940,"thread_start_us":2860,"threads_started":1}
I20260924 11:47:25.654862 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.643079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:40567 (local address 127.0.0.1:41116)
0924 11:47:25.645530 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.645546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.645647 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.646961 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.646965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.646996 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:25.647449 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.647458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.649268 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.649272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.651766 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.651780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.653989 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.653997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.654209 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.654238 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.654320 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.654463 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2330,"mutex_wait_us":31,"thread_start_us":716,"threads_started":1}
I20260924 11:47:25.654973 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.644180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.1.210.1:39787)
0924 11:47:25.644710 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.644721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.646975 (+ 2254us) server_negotiation.cc:407] Connection header received
0924 11:47:25.647247 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.647257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.647479 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.647669 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:25.648199 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.649586 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.652140 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.653571 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.653694 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.653880 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.654472 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.654669 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260924 11:47:25.655809 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.643645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.1.210.2:40567)
0924 11:47:25.646455 (+ 2810us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.646462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.646484 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:47:25.646564 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.646570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.646768 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.646961 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:25.647592 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.649137 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.651968 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.653515 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.653631 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.653817 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.655419 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.655554 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2678,"thread_start_us":2568,"threads_started":1}
I20260924 11:47:25.654907 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.643705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39787 (local address 127.0.0.1:54714)
0924 11:47:25.646850 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.646864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.647071 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.647648 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.647655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.647681 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:25.648013 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.648024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.649725 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.649734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.651945 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.651956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.654043 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.654049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.654265 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.654306 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.654407 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.654535 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3029,"mutex_wait_us":6,"thread_start_us":1546,"threads_started":1}
I20260924 11:47:25.657806 4030 tablet_service.cc:1467] Processing CreateTablet for tablet e864d3050d5d487a99a5d2159261b7cf (DEFAULT_TABLE table=test-table [id=c2f67b9c022446699b34d7e477878526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:25.659272 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e864d3050d5d487a99a5d2159261b7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:25.658962 3891 tablet_service.cc:1467] Processing CreateTablet for tablet e864d3050d5d487a99a5d2159261b7cf (DEFAULT_TABLE table=test-table [id=c2f67b9c022446699b34d7e477878526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:25.660216 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e864d3050d5d487a99a5d2159261b7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:25.668833 3752 tablet_service.cc:1467] Processing CreateTablet for tablet e864d3050d5d487a99a5d2159261b7cf (DEFAULT_TABLE table=test-table [id=c2f67b9c022446699b34d7e477878526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:25.670168 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e864d3050d5d487a99a5d2159261b7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:25.674582 4120 tablet_bootstrap.cc:492] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Bootstrap starting.
I20260924 11:47:25.674851 4119 tablet_bootstrap.cc:492] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Bootstrap starting.
I20260924 11:47:25.678419 4119 tablet_bootstrap.cc:654] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:25.679754 4119 log.cc:824] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:25.679899 4120 tablet_bootstrap.cc:654] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:25.681164 4120 log.cc:824] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:25.682698 4119 tablet_bootstrap.cc:492] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: No bootstrap required, opened a new log
I20260924 11:47:25.682966 4119 ts_tablet_manager.cc:1397] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:47:25.684077 4123 tablet_bootstrap.cc:492] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Bootstrap starting.
I20260924 11:47:25.684124 4120 tablet_bootstrap.cc:492] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: No bootstrap required, opened a new log
I20260924 11:47:25.684352 4120 ts_tablet_manager.cc:1397] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260924 11:47:25.687757 4123 tablet_bootstrap.cc:654] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:25.688966 4123 log.cc:824] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:25.691161 4119 raft_consensus.cc:348] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.691515 4119 raft_consensus.cc:374] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:25.691617 4119 raft_consensus.cc:729] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Initialized, Role: FOLLOWER
I20260924 11:47:25.691807 4123 tablet_bootstrap.cc:492] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: No bootstrap required, opened a new log
I20260924 11:47:25.692049 4123 ts_tablet_manager.cc:1397] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 11:47:25.692309 4119 consensus_queue.cc:260] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.693352 3961 heartbeater.cc:503] Master 127.1.210.62:33025 was elected leader, sending a full tablet report...
I20260924 11:47:25.694510 4120 raft_consensus.cc:348] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.694967 4120 raft_consensus.cc:374] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:25.695096 4120 raft_consensus.cc:729] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdecbf8093944cc83fa9ce02ca2a83c, State: Initialized, Role: FOLLOWER
I20260924 11:47:25.695191 4119 ts_tablet_manager.cc:1428] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 11:47:25.695771 4120 consensus_queue.cc:260] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.696859 4100 heartbeater.cc:503] Master 127.1.210.62:33025 was elected leader, sending a full tablet report...
I20260924 11:47:25.697412 4120 ts_tablet_manager.cc:1428] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260924 11:47:25.701705 4123 raft_consensus.cc:348] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.702034 4123 raft_consensus.cc:374] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:25.702109 4123 raft_consensus.cc:729] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8be1202d2fc4120a41b27fc188d96f1, State: Initialized, Role: FOLLOWER
I20260924 11:47:25.702651 4123 consensus_queue.cc:260] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.704288 4123 ts_tablet_manager.cc:1428] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 11:47:25.752491 4125 raft_consensus.cc:482] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:25.752794 4125 raft_consensus.cc:504] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.754467 4125 leader_election.cc:288] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253), f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787)
I20260924 11:47:25.761196 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.754993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:40695 (local address 127.1.210.1:39787)
0924 11:47:25.755227 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.755236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.756001 (+ 765us) server_negotiation.cc:407] Connection header received
0924 11:47:25.756188 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.756192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.756258 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.756356 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:25.756991 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.757735 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.759441 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.760213 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.760366 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.760482 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.760780 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.760901 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:47:25.761194 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.754804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:40767 (local address 127.1.210.3:36253)
0924 11:47:25.755025 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.755032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.755895 (+ 863us) server_negotiation.cc:407] Connection header received
0924 11:47:25.756083 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.756087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.756143 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.756243 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:25.756816 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.757594 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.759393 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.760147 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.760308 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.760439 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.760780 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.760900 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:47:25.761387 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.754897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:36253 (local address 127.1.210.2:40767)
0924 11:47:25.755779 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.755794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.755911 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.756257 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.756262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.756283 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:25.756666 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.756675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.757725 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.757731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.759241 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.759252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.760402 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.760411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.760580 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.760651 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.760755 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.760885 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":101,"thread_start_us":531,"threads_started":1}
I20260924 11:47:25.761422 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.754936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39787 (local address 127.1.210.2:40695)
0924 11:47:25.755879 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.755898 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.756040 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.756367 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.756371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.756407 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:25.756845 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.756853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.757870 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.757879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.759287 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.759298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.760449 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.760455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.760633 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.760655 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.760756 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.760888 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":44,"thread_start_us":508,"threads_started":1}
I20260924 11:47:25.762389 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e864d3050d5d487a99a5d2159261b7cf" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" is_pre_election: true
I20260924 11:47:25.762442 3776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e864d3050d5d487a99a5d2159261b7cf" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1" is_pre_election: true
I20260924 11:47:25.762864 4050 raft_consensus.cc:2454] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:25.762883 3776 raft_consensus.cc:2454] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:25.763456 3843 leader_election.cc:302] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, ccdecbf8093944cc83fa9ce02ca2a83c; no voters:
I20260924 11:47:25.763830 4125 raft_consensus.cc:2781] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:25.763976 4125 raft_consensus.cc:482] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:25.764072 4125 raft_consensus.cc:3037] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:25.766225 4125 raft_consensus.cc:504] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.766826 4125 leader_election.cc:288] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Requested vote from peers ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253), f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787)
I20260924 11:47:25.767143 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e864d3050d5d487a99a5d2159261b7cf" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
I20260924 11:47:25.767321 3776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e864d3050d5d487a99a5d2159261b7cf" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1"
I20260924 11:47:25.767362 4050 raft_consensus.cc:3037] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:25.767531 3776 raft_consensus.cc:3037] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:25.770641 3776 raft_consensus.cc:2454] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:25.770641 4050 raft_consensus.cc:2454] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:25.771140 3845 leader_election.cc:302] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, ccdecbf8093944cc83fa9ce02ca2a83c, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:25.771716 4125 raft_consensus.cc:2781] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:25.772233 4125 raft_consensus.cc:686] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Becoming Leader. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Running, Role: LEADER
I20260924 11:47:25.772806 4125 consensus_queue.cc:237] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } }
I20260924 11:47:25.777345 3613 catalog_manager.cc:5495] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "7ffd872059bf4404bc6927f8d4790f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } health_report { overall_health: UNKNOWN } } }
W20260924 11:47:25.793263 3823 tablet.cc:2369] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:47:25.803074 3962 tablet.cc:2369] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:47:25.814064 4101 tablet.cc:2369] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 11:47:25 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250443, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:25.840023 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.827456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.1.210.62:33025)
0924 11:47:25.827690 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:25.827697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:25.828180 (+ 483us) server_negotiation.cc:407] Connection header received
0924 11:47:25.828313 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:25.828316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:25.828373 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:25.828467 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:25.829632 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.830324 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.831547 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.831996 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.832102 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.835262 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:25.835266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:25.835269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:25.835295 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:25.837846 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.838437 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.838441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.838443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.838560 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:25.838884 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:25.838887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:25.838889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:25.839113 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:25.839229 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:25.839564 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 11:47:25.839768 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":75}
I20260924 11:47:25.840256 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:25.827456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33025 (local address 127.0.0.1:37628)
0924 11:47:25.828076 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:25.828092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:25.828215 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:25.828508 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:25.828511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:25.829242 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:25.829503 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.829512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.830432 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.830436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.831392 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:25.831402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:25.832152 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:25.832157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:25.832262 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:25.832839 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:25.832854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:25.835111 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:25.837958 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.837966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.837972 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.838334 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.838650 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:25.838653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:25.838656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:25.838798 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:25.839217 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:25.839222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:25.839342 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 11:47:25.839483 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 11:47:25.839659 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":362,"threads_started":1}
I20260924 11:47:25.844897 3613 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:37628:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:47:25.845903 3613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:25.851948 3752 tablet_service.cc:1467] Processing CreateTablet for tablet 22d7550924f94a4db4f912560beb055a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b22a85087c6f4a96ad1f528f7ad8290e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:47:25.852120 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 22d7550924f94a4db4f912560beb055a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b22a85087c6f4a96ad1f528f7ad8290e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:47:25.852931 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d7550924f94a4db4f912560beb055a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:25.853144 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d7550924f94a4db4f912560beb055a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:25.852320 4030 tablet_service.cc:1467] Processing CreateTablet for tablet 22d7550924f94a4db4f912560beb055a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b22a85087c6f4a96ad1f528f7ad8290e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:47:25.853480 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d7550924f94a4db4f912560beb055a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:25.858180 4119 tablet_bootstrap.cc:492] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: Bootstrap starting.
I20260924 11:47:25.859479 4119 tablet_bootstrap.cc:654] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:25.859906 4123 tablet_bootstrap.cc:492] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: Bootstrap starting.
I20260924 11:47:25.860121 4120 tablet_bootstrap.cc:492] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: Bootstrap starting.
I20260924 11:47:25.861512 4123 tablet_bootstrap.cc:654] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:25.861657 4120 tablet_bootstrap.cc:654] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:25.861922 4119 tablet_bootstrap.cc:492] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: No bootstrap required, opened a new log
I20260924 11:47:25.862067 4119 ts_tablet_manager.cc:1397] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:25.862855 4119 raft_consensus.cc:348] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:25.863125 4119 raft_consensus.cc:374] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:25.863214 4119 raft_consensus.cc:729] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Initialized, Role: FOLLOWER
I20260924 11:47:25.863499 4119 consensus_queue.cc:260] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:25.864053 4123 tablet_bootstrap.cc:492] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: No bootstrap required, opened a new log
I20260924 11:47:25.863907 4125 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:25.864195 4125 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:25.864188 4123 ts_tablet_manager.cc:1397] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:47:25.864379 4120 tablet_bootstrap.cc:492] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: No bootstrap required, opened a new log
I20260924 11:47:25.864488 4120 ts_tablet_manager.cc:1397] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:47:25.864910 4123 raft_consensus.cc:348] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:25.865175 4123 raft_consensus.cc:374] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:25.865271 4123 raft_consensus.cc:729] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8be1202d2fc4120a41b27fc188d96f1, State: Initialized, Role: FOLLOWER
I20260924 11:47:25.865149 4120 raft_consensus.cc:348] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:25.865372 4120 raft_consensus.cc:374] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:25.865476 4120 raft_consensus.cc:729] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdecbf8093944cc83fa9ce02ca2a83c, State: Initialized, Role: FOLLOWER
I20260924 11:47:25.865486 4123 consensus_queue.cc:260] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:25.865726 4119 ts_tablet_manager.cc:1428] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:25.865679 4120 consensus_queue.cc:260] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:25.866046 4123 ts_tablet_manager.cc:1428] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:47:25.866117 4126 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:25.866485 4126 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:25.866505 4127 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:25.866928 4127 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:25.866958 4120 ts_tablet_manager.cc:1428] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:47:26.168794 4127 raft_consensus.cc:482] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:26.169085 4127 raft_consensus.cc:504] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.170567 4127 leader_election.cc:288] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:26.177575 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.171020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:42145 (local address 127.1.210.3:36253)
0924 11:47:26.171242 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:26.171252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:26.172101 (+ 849us) server_negotiation.cc:407] Connection header received
0924 11:47:26.172316 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:26.172320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:26.172387 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:26.172498 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:26.173232 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.173950 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.175933 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.176639 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.176790 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.176895 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:26.177210 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 11:47:26.177331 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:47:26.177716 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.170957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:58753 (local address 127.1.210.2:40567)
0924 11:47:26.171623 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:26.171631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:26.172253 (+ 622us) server_negotiation.cc:407] Connection header received
0924 11:47:26.172485 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:26.172489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:26.172543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:26.172646 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:26.173338 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.174047 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.175927 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.176676 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.176831 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.176943 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:26.177274 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 11:47:26.177401 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":440,"threads_started":1}
I20260924 11:47:26.177963 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.170923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:36253 (local address 127.1.210.1:42145)
0924 11:47:26.171965 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:26.171988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:26.172154 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:26.172528 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:26.172533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:26.172559 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:26.173026 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.173048 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.174096 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.174101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.175776 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.175787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.176864 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.176871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.177030 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.177052 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:47:26.177177 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 11:47:26.177295 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":96,"thread_start_us":694,"threads_started":1}
I20260924 11:47:26.178062 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.170974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:40567 (local address 127.1.210.1:58753)
0924 11:47:26.172172 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:26.172186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:26.172345 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:26.172663 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:26.172666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:26.172704 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:26.173184 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.173193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.174250 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.174255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.175781 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.175792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.176905 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.176911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.177130 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.177154 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 11:47:26.177250 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:47:26.177367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":23,"thread_start_us":826,"threads_started":1}
I20260924 11:47:26.178614 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d7550924f94a4db4f912560beb055a" candidate_uuid: "f8be1202d2fc4120a41b27fc188d96f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" is_pre_election: true
I20260924 11:47:26.178861 4050 raft_consensus.cc:2454] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8be1202d2fc4120a41b27fc188d96f1 in term 0.
I20260924 11:47:26.178900 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d7550924f94a4db4f912560beb055a" candidate_uuid: "f8be1202d2fc4120a41b27fc188d96f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ffd872059bf4404bc6927f8d4790f07" is_pre_election: true
I20260924 11:47:26.179349 3911 raft_consensus.cc:2454] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8be1202d2fc4120a41b27fc188d96f1 in term 0.
I20260924 11:47:26.179340 3703 leader_election.cc:302] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccdecbf8093944cc83fa9ce02ca2a83c, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:26.179774 4127 raft_consensus.cc:2781] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:26.179908 4127 raft_consensus.cc:482] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:26.180042 4127 raft_consensus.cc:3037] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:26.182236 4127 raft_consensus.cc:504] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.182885 4127 leader_election.cc:288] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [CANDIDATE]: Term 1 election: Requested vote from peers 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:26.183354 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d7550924f94a4db4f912560beb055a" candidate_uuid: "f8be1202d2fc4120a41b27fc188d96f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
I20260924 11:47:26.183331 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d7550924f94a4db4f912560beb055a" candidate_uuid: "f8be1202d2fc4120a41b27fc188d96f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ffd872059bf4404bc6927f8d4790f07"
I20260924 11:47:26.183550 3911 raft_consensus.cc:3037] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:26.183563 4050 raft_consensus.cc:3037] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:26.186489 4050 raft_consensus.cc:2454] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8be1202d2fc4120a41b27fc188d96f1 in term 1.
I20260924 11:47:26.186523 3911 raft_consensus.cc:2454] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8be1202d2fc4120a41b27fc188d96f1 in term 1.
I20260924 11:47:26.186932 3703 leader_election.cc:302] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccdecbf8093944cc83fa9ce02ca2a83c, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:26.187227 4127 raft_consensus.cc:2781] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:26.187733 4127 raft_consensus.cc:686] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 LEADER]: Becoming Leader. State: Replica: f8be1202d2fc4120a41b27fc188d96f1, State: Running, Role: LEADER
I20260924 11:47:26.188275 4127 consensus_queue.cc:237] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.191465 4144 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: TxnStatusTablet state changed. Reason: New leader f8be1202d2fc4120a41b27fc188d96f1. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.191893 4144 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:47:26.192519 4146 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:47:26.192767 4146 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:47:26.192678 3615 catalog_manager.cc:5495] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1). New cstate: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:26.203508 4125 consensus_queue.cc:1035] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:26.210148 4125 consensus_queue.cc:1035] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:26.238078 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.231993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.1.210.1:39787)
0924 11:47:26.232164 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:26.232169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:26.232264 (+ 95us) server_negotiation.cc:407] Connection header received
0924 11:47:26.232545 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:26.232549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:26.232637 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:26.232746 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:26.233179 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.233996 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.235916 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.236314 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.236405 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.236952 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.237521 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.237623 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:26.237769 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 11:47:26.237833 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:47:26.238198 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.231916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39787 (local address 127.0.0.1:54728)
0924 11:47:26.232170 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:26.232190 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:26.232339 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:26.232799 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:26.232804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:26.232817 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:47:26.233029 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.233040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.234123 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.234129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.235787 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.235801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.236496 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.236502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.236688 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.236759 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.237630 (+ 871us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.237651 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:47:26.237746 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:47:26.237904 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:47:26.245793 4050 raft_consensus.cc:1261] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Refusing update from remote peer f8be1202d2fc4120a41b27fc188d96f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:26.245908 3911 raft_consensus.cc:1261] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Refusing update from remote peer f8be1202d2fc4120a41b27fc188d96f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:26.246551 4144 consensus_queue.cc:1035] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:26.247180 4127 consensus_queue.cc:1035] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:26.257931 4125 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: TxnStatusTablet state changed. Reason: New leader f8be1202d2fc4120a41b27fc188d96f1. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.258250 4125 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:26.259146 4126 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: TxnStatusTablet state changed. Reason: New leader f8be1202d2fc4120a41b27fc188d96f1. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.259413 4126 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:26.262586 4150 mvcc.cc:205] Tried to move back new op lower bound from 7332865827814924288 to 7332865827590897664. Current Snapshot: MvccSnapshot[applied={T|T < 7332865827814924288}]
I20260924 11:47:26.263114 4152 mvcc.cc:205] Tried to move back new op lower bound from 7332865827814924288 to 7332865827590897664. Current Snapshot: MvccSnapshot[applied={T|T < 7332865827814924288}]
I20260924 11:47:26.265326 4126 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.265571 4126 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P ccdecbf8093944cc83fa9ce02ca2a83c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:26.266312 4144 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.266578 4144 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:47:26.266844 4144 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.267081 4144 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P f8be1202d2fc4120a41b27fc188d96f1: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:47:26.275290 4125 tablet_replica.cc:439] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8be1202d2fc4120a41b27fc188d96f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } } }
I20260924 11:47:26.275580 4125 tablet_replica.cc:442] T 22d7550924f94a4db4f912560beb055a P 7ffd872059bf4404bc6927f8d4790f07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:47:26.284650 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.278333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.1.210.2:40567)
0924 11:47:26.278526 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:26.278532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:26.279006 (+ 474us) server_negotiation.cc:407] Connection header received
0924 11:47:26.279208 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:26.279215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:26.279323 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:26.279509 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:26.279889 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.281014 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.282373 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.283015 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.283129 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.283501 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.284114 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.284251 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:26.284297 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 11:47:26.284371 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:47:26.300268 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.294189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39787 (local address 127.1.210.2:51251)
0924 11:47:26.294709 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:26.294728 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:26.294865 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:26.295230 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:26.295233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:26.295242 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:47:26.295526 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.295534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.296979 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.296982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.298244 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.298256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.299230 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.299235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.299346 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.299393 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.299794 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.299801 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:47:26.299858 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 11:47:26.300011 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":690,"threads_started":1}
I20260924 11:47:26.300268 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.294328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:51251 (local address 127.1.210.1:39787)
0924 11:47:26.294551 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:26.294557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:26.294815 (+ 258us) server_negotiation.cc:407] Connection header received
0924 11:47:26.295033 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:26.295039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:26.295118 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:26.295219 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:26.295664 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.296860 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.298400 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.299090 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.299217 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.299575 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.299700 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.299788 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:26.299904 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 11:47:26.299988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:47:26.343171 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.336608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:40567 (local address 127.1.210.1:56787)
0924 11:47:26.337319 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:26.337344 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:26.337534 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:26.338013 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:26.338018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:26.338030 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:47:26.338439 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.338451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.339596 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.339600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.340961 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:26.340974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.341836 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.341843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:26.342071 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.342146 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.342584 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.342593 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 11:47:26.342664 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 11:47:26.342765 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":612,"threads_started":1}
I20260924 11:47:26.343194 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:26.336590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:56787 (local address 127.1.210.2:40567)
0924 11:47:26.337037 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:26.337043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:26.337415 (+ 372us) server_negotiation.cc:407] Connection header received
0924 11:47:26.337664 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:26.337671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:26.337759 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:26.337841 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:26.338630 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.339452 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.341116 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:26.341644 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:26.341781 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:26.342332 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:26.342480 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:26.342567 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:26.342680 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 11:47:26.342761 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260924 11:47:26.381456 3615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:26.381742 3615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:26.386087 3615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Sending DeleteTablet for 3 replicas of tablet e864d3050d5d487a99a5d2159261b7cf
I20260924 11:47:26.387668 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet e864d3050d5d487a99a5d2159261b7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:26 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:41116
I20260924 11:47:26.387916 3752 tablet_service.cc:1514] Processing DeleteTablet for tablet e864d3050d5d487a99a5d2159261b7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:26 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260924 11:47:26.390376 4030 tablet_service.cc:1514] Processing DeleteTablet for tablet e864d3050d5d487a99a5d2159261b7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:26 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260924 11:47:26.390353 3615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:26.391278 3615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:26.400691 3890 tablet_service.cc:1467] Processing CreateTablet for tablet 796e79026f4943dca5adc3b269ab7cf0 (DEFAULT_TABLE table=test-table [id=2902be901e3942178bff415dbe975caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:26.401320 3890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796e79026f4943dca5adc3b269ab7cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:26.406118 3751 tablet_service.cc:1467] Processing CreateTablet for tablet 796e79026f4943dca5adc3b269ab7cf0 (DEFAULT_TABLE table=test-table [id=2902be901e3942178bff415dbe975caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:26.406615 3751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796e79026f4943dca5adc3b269ab7cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:26.406615 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 796e79026f4943dca5adc3b269ab7cf0 (DEFAULT_TABLE table=test-table [id=2902be901e3942178bff415dbe975caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:26.407076 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796e79026f4943dca5adc3b269ab7cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:26.413618 4173 tablet_replica.cc:330] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: stopping tablet replica
I20260924 11:47:26.414083 4173 raft_consensus.cc:2229] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:26.414492 4173 raft_consensus.cc:2258] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:26.416429 4173 ts_tablet_manager.cc:1905] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:26.426548 4174 tablet_bootstrap.cc:492] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Bootstrap starting.
I20260924 11:47:26.428501 4174 tablet_bootstrap.cc:654] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:26.429276 4173 ts_tablet_manager.cc:1918] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:26.429976 4176 tablet_bootstrap.cc:492] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Bootstrap starting.
I20260924 11:47:26.432323 4176 tablet_bootstrap.cc:654] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:26.432523 4172 tablet_replica.cc:330] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: stopping tablet replica
I20260924 11:47:26.432576 4173 log.cc:1196] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/e864d3050d5d487a99a5d2159261b7cf
I20260924 11:47:26.433090 4172 raft_consensus.cc:2229] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:26.433362 4171 tablet_replica.cc:330] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: stopping tablet replica
I20260924 11:47:26.433612 4172 raft_consensus.cc:2258] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:26.434149 4171 raft_consensus.cc:2229] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:26.434904 4171 raft_consensus.cc:2258] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:26.435127 4175 tablet_bootstrap.cc:492] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Bootstrap starting.
I20260924 11:47:26.437006 4175 tablet_bootstrap.cc:654] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:26.437166 4171 ts_tablet_manager.cc:1905] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:26.438571 4172 ts_tablet_manager.cc:1905] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:26.440347 4174 tablet_bootstrap.cc:492] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: No bootstrap required, opened a new log
I20260924 11:47:26.440557 4174 ts_tablet_manager.cc:1397] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260924 11:47:26.441412 4174 raft_consensus.cc:348] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.441751 4174 raft_consensus.cc:374] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:26.441884 4174 raft_consensus.cc:729] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdecbf8093944cc83fa9ce02ca2a83c, State: Initialized, Role: FOLLOWER
I20260924 11:47:26.442250 4174 consensus_queue.cc:260] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.444916 4171 ts_tablet_manager.cc:1918] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:26.445411 4171 log.cc:1196] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/e864d3050d5d487a99a5d2159261b7cf
I20260924 11:47:26.445992 4171 ts_tablet_manager.cc:1939] T e864d3050d5d487a99a5d2159261b7cf P 7ffd872059bf4404bc6927f8d4790f07: Deleting consensus metadata
I20260924 11:47:26.446285 4174 ts_tablet_manager.cc:1428] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 11:47:26.447664 3599 catalog_manager.cc:4841] TS 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567): tablet e864d3050d5d487a99a5d2159261b7cf (table test-table [id=c2f67b9c022446699b34d7e477878526]) successfully deleted
I20260924 11:47:26.449239 4175 tablet_bootstrap.cc:492] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: No bootstrap required, opened a new log
I20260924 11:47:26.449450 4175 ts_tablet_manager.cc:1397] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260924 11:47:26.450029 4172 ts_tablet_manager.cc:1918] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:26.450265 4173 ts_tablet_manager.cc:1939] T e864d3050d5d487a99a5d2159261b7cf P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting consensus metadata
I20260924 11:47:26.451871 3600 catalog_manager.cc:4841] TS ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253): tablet e864d3050d5d487a99a5d2159261b7cf (table test-table [id=c2f67b9c022446699b34d7e477878526]) successfully deleted
I20260924 11:47:26.450246 4175 raft_consensus.cc:348] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.452184 4175 raft_consensus.cc:374] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:26.452296 4175 raft_consensus.cc:729] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8be1202d2fc4120a41b27fc188d96f1, State: Initialized, Role: FOLLOWER
I20260924 11:47:26.452205 4172 log.cc:1196] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/e864d3050d5d487a99a5d2159261b7cf
I20260924 11:47:26.452540 4175 consensus_queue.cc:260] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.452922 4172 ts_tablet_manager.cc:1939] T e864d3050d5d487a99a5d2159261b7cf P f8be1202d2fc4120a41b27fc188d96f1: Deleting consensus metadata
I20260924 11:47:26.453663 4176 tablet_bootstrap.cc:492] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: No bootstrap required, opened a new log
I20260924 11:47:26.453814 4176 ts_tablet_manager.cc:1397] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20260924 11:47:26.455250 4176 raft_consensus.cc:348] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.455681 3602 catalog_manager.cc:4841] TS f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787): tablet e864d3050d5d487a99a5d2159261b7cf (table test-table [id=c2f67b9c022446699b34d7e477878526]) successfully deleted
I20260924 11:47:26.455891 4176 raft_consensus.cc:374] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:26.453676 4175 ts_tablet_manager.cc:1428] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:26.456087 4176 raft_consensus.cc:729] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Initialized, Role: FOLLOWER
I20260924 11:47:26.456465 4176 consensus_queue.cc:260] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.457901 4176 ts_tablet_manager.cc:1428] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:26.513971 4130 raft_consensus.cc:482] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:26.514212 4130 raft_consensus.cc:504] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.515009 4130 leader_election.cc:288] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:26.515473 3776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e79026f4943dca5adc3b269ab7cf0" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1" is_pre_election: true
I20260924 11:47:26.515741 3776 raft_consensus.cc:2454] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:26.515707 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e79026f4943dca5adc3b269ab7cf0" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" is_pre_election: true
I20260924 11:47:26.515946 4050 raft_consensus.cc:2454] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:26.516170 3845 leader_election.cc:302] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:26.516561 4130 raft_consensus.cc:2781] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:26.516676 4130 raft_consensus.cc:482] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:26.516768 4130 raft_consensus.cc:3037] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:26.520110 4130 raft_consensus.cc:504] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.520778 4130 leader_election.cc:288] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Requested vote from peers f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:26.521253 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e79026f4943dca5adc3b269ab7cf0" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
I20260924 11:47:26.521399 3776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796e79026f4943dca5adc3b269ab7cf0" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1"
I20260924 11:47:26.521476 4050 raft_consensus.cc:3037] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:26.521608 3776 raft_consensus.cc:3037] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:26.524298 4050 raft_consensus.cc:2454] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:26.524744 3843 leader_election.cc:302] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, ccdecbf8093944cc83fa9ce02ca2a83c; no voters:
I20260924 11:47:26.524771 3776 raft_consensus.cc:2454] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:26.525318 4130 raft_consensus.cc:2781] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:26.525569 4130 raft_consensus.cc:686] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Becoming Leader. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Running, Role: LEADER
I20260924 11:47:26.525897 4130 consensus_queue.cc:237] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:26.528877 3614 catalog_manager.cc:5495] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "7ffd872059bf4404bc6927f8d4790f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:26.570149 4049 raft_consensus.cc:1261] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Refusing update from remote peer 7ffd872059bf4404bc6927f8d4790f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:26.570726 3776 raft_consensus.cc:1261] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Refusing update from remote peer 7ffd872059bf4404bc6927f8d4790f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:26.571796 4130 consensus_queue.cc:1035] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:26.572996 4130 consensus_queue.cc:1035] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:26.584654 4149 mvcc.cc:205] Tried to move back new op lower bound from 7332865829145980928 to 7332865828973096960. Current Snapshot: MvccSnapshot[applied={T|T < 7332865829145980928}]
W20260924 11:47:26.602568 4149 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 11:47:28.198406 3615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:28.198613 3615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:28.202373 3615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Sending DeleteTablet for 3 replicas of tablet 796e79026f4943dca5adc3b269ab7cf0
I20260924 11:47:28.203230 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet 796e79026f4943dca5adc3b269ab7cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:28 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:41116
I20260924 11:47:28.203329 3751 tablet_service.cc:1514] Processing DeleteTablet for tablet 796e79026f4943dca5adc3b269ab7cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:28 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260924 11:47:28.203523 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet 796e79026f4943dca5adc3b269ab7cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:28 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260924 11:47:28.215857 4207 tablet_replica.cc:330] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: stopping tablet replica
I20260924 11:47:28.215896 4208 tablet_replica.cc:330] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: stopping tablet replica
I20260924 11:47:28.216290 4207 raft_consensus.cc:2229] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:28.216292 4208 raft_consensus.cc:2229] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:28.216665 4208 raft_consensus.cc:2258] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:28.216673 4207 raft_consensus.cc:2258] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:28.217303 4206 tablet_replica.cc:330] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: stopping tablet replica
I20260924 11:47:28.217556 4206 raft_consensus.cc:2229] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:28.217777 4207 ts_tablet_manager.cc:1905] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:28.217962 4206 raft_consensus.cc:2258] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:28.218040 4208 ts_tablet_manager.cc:1905] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:28.219978 4206 ts_tablet_manager.cc:1905] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:28.225625 4208 ts_tablet_manager.cc:1918] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:47:28.225788 4207 ts_tablet_manager.cc:1918] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:47:28.225834 4208 log.cc:1196] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/796e79026f4943dca5adc3b269ab7cf0
I20260924 11:47:28.226007 4207 log.cc:1196] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/796e79026f4943dca5adc3b269ab7cf0
I20260924 11:47:28.226294 4208 ts_tablet_manager.cc:1939] T 796e79026f4943dca5adc3b269ab7cf0 P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting consensus metadata
I20260924 11:47:28.226528 4207 ts_tablet_manager.cc:1939] T 796e79026f4943dca5adc3b269ab7cf0 P f8be1202d2fc4120a41b27fc188d96f1: Deleting consensus metadata
I20260924 11:47:28.227461 3600 catalog_manager.cc:4841] TS ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253): tablet 796e79026f4943dca5adc3b269ab7cf0 (table test-table [id=2902be901e3942178bff415dbe975caa]) successfully deleted
I20260924 11:47:28.227684 3602 catalog_manager.cc:4841] TS f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787): tablet 796e79026f4943dca5adc3b269ab7cf0 (table test-table [id=2902be901e3942178bff415dbe975caa]) successfully deleted
I20260924 11:47:28.231472 4206 ts_tablet_manager.cc:1918] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:47:28.231626 4206 log.cc:1196] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/796e79026f4943dca5adc3b269ab7cf0
I20260924 11:47:28.232002 4206 ts_tablet_manager.cc:1939] T 796e79026f4943dca5adc3b269ab7cf0 P 7ffd872059bf4404bc6927f8d4790f07: Deleting consensus metadata
I20260924 11:47:28.232944 3599 catalog_manager.cc:4841] TS 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567): tablet 796e79026f4943dca5adc3b269ab7cf0 (table test-table [id=2902be901e3942178bff415dbe975caa]) successfully deleted
Sep 24 11:47:28 dist-test-slave-dd32 krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:47:28 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:28.260756 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:28.246852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.1.210.62:33025)
0924 11:47:28.247406 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:28.247410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:28.247435 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:47:28.247641 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:28.247645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:28.247716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:28.247809 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:28.249028 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:28.250266 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:28.251460 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:28.251858 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:28.251951 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:28.255457 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:28.255462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:28.255465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:28.255493 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:28.258510 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:28.259189 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:28.259193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:28.259195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:28.259298 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:28.259598 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:28.259601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:28.259603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:28.259834 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:28.260008 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:28.260230 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:47:28.260372 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":301,"threads_started":1}
I20260924 11:47:28.263818 3615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:28.264412 3615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:28.270509 3751 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3ea950f1042898f673ffbd6e1eb0a (DEFAULT_TABLE table=test-table [id=082e1956d0564aad8c13bb4bdc614014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:28.270767 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3ea950f1042898f673ffbd6e1eb0a (DEFAULT_TABLE table=test-table [id=082e1956d0564aad8c13bb4bdc614014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:28.270865 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3ea950f1042898f673ffbd6e1eb0a (DEFAULT_TABLE table=test-table [id=082e1956d0564aad8c13bb4bdc614014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:28.270985 3751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3ea950f1042898f673ffbd6e1eb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:28.271066 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3ea950f1042898f673ffbd6e1eb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:28.271268 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3ea950f1042898f673ffbd6e1eb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:28.276325 4217 tablet_bootstrap.cc:492] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Bootstrap starting.
I20260924 11:47:28.277789 4217 tablet_bootstrap.cc:654] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:28.278405 4219 tablet_bootstrap.cc:492] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Bootstrap starting.
I20260924 11:47:28.279207 4218 tablet_bootstrap.cc:492] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Bootstrap starting.
I20260924 11:47:28.280323 4219 tablet_bootstrap.cc:654] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:28.280865 4217 tablet_bootstrap.cc:492] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: No bootstrap required, opened a new log
I20260924 11:47:28.281014 4218 tablet_bootstrap.cc:654] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:28.281462 4217 ts_tablet_manager.cc:1397] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 11:47:28.282372 4217 raft_consensus.cc:348] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.282640 4217 raft_consensus.cc:374] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:28.282771 4217 raft_consensus.cc:729] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Initialized, Role: FOLLOWER
I20260924 11:47:28.283092 4219 tablet_bootstrap.cc:492] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: No bootstrap required, opened a new log
I20260924 11:47:28.283021 4217 consensus_queue.cc:260] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.283249 4218 tablet_bootstrap.cc:492] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: No bootstrap required, opened a new log
I20260924 11:47:28.283236 4219 ts_tablet_manager.cc:1397] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 11:47:28.283321 4184 raft_consensus.cc:482] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:28.283610 4217 ts_tablet_manager.cc:1428] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:47:28.283468 4184 raft_consensus.cc:504] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.284137 4184 leader_election.cc:288] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:28.284027 4219 raft_consensus.cc:348] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.284248 4219 raft_consensus.cc:374] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:28.284340 4219 raft_consensus.cc:729] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8be1202d2fc4120a41b27fc188d96f1, State: Initialized, Role: FOLLOWER
I20260924 11:47:28.284525 4218 ts_tablet_manager.cc:1397] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 11:47:28.284571 4219 consensus_queue.cc:260] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.285138 4219 ts_tablet_manager.cc:1428] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:47:28.285305 4218 raft_consensus.cc:348] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.285565 4218 raft_consensus.cc:374] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:28.285538 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3ea950f1042898f673ffbd6e1eb0a" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" is_pre_election: true
I20260924 11:47:28.285666 4218 raft_consensus.cc:729] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdecbf8093944cc83fa9ce02ca2a83c, State: Initialized, Role: FOLLOWER
I20260924 11:47:28.285948 3773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3ea950f1042898f673ffbd6e1eb0a" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1" is_pre_election: true
I20260924 11:47:28.285908 4218 consensus_queue.cc:260] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.286172 3773 raft_consensus.cc:2454] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:28.286707 4218 ts_tablet_manager.cc:1428] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 11:47:28.286659 3845 leader_election.cc:302] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:28.286975 4201 raft_consensus.cc:2781] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:28.287159 4201 raft_consensus.cc:482] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:28.287284 4201 raft_consensus.cc:3037] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:28.289850 4201 raft_consensus.cc:504] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.290902 4201 leader_election.cc:288] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Requested vote from peers f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:28.291030 3773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3ea950f1042898f673ffbd6e1eb0a" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1"
I20260924 11:47:28.291105 4050 raft_consensus.cc:2454] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:28.291245 3773 raft_consensus.cc:3037] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:28.293321 3773 raft_consensus.cc:2454] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:28.293254 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3ea950f1042898f673ffbd6e1eb0a" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
I20260924 11:47:28.293503 4049 raft_consensus.cc:3037] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:28.293774 3845 leader_election.cc:302] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:28.294023 4201 raft_consensus.cc:2781] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:28.294194 4201 raft_consensus.cc:686] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Becoming Leader. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Running, Role: LEADER
I20260924 11:47:28.294407 4201 consensus_queue.cc:237] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.296093 4049 raft_consensus.cc:2454] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:28.297858 3615 catalog_manager.cc:5495] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "7ffd872059bf4404bc6927f8d4790f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:28.322494 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:28.317530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.1.210.2:40567)
0924 11:47:28.318039 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:28.318043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:28.318062 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:47:28.318113 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:28.318117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:28.318172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:28.318253 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:28.318617 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:28.319532 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:28.320805 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:28.321239 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:28.321339 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:28.321747 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:28.321852 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:28.321922 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:28.322059 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 11:47:28.322120 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":311,"threads_started":1}
I20260924 11:47:28.331153 4049 raft_consensus.cc:1261] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Refusing update from remote peer 7ffd872059bf4404bc6927f8d4790f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:28.331360 3773 raft_consensus.cc:1261] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Refusing update from remote peer 7ffd872059bf4404bc6927f8d4790f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:28.331715 4201 consensus_queue.cc:1035] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:28.331975 4184 consensus_queue.cc:1035] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:28.373304 3614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:28.373554 3614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:28.376152 3614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Sending DeleteTablet for 3 replicas of tablet 26f3ea950f1042898f673ffbd6e1eb0a
I20260924 11:47:28.376798 3751 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f3ea950f1042898f673ffbd6e1eb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:28 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260924 11:47:28.376865 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f3ea950f1042898f673ffbd6e1eb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:28 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:41116
I20260924 11:47:28.376928 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f3ea950f1042898f673ffbd6e1eb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:28 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260924 11:47:28.377130 4206 tablet_replica.cc:330] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: stopping tablet replica
I20260924 11:47:28.377172 4207 tablet_replica.cc:330] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: stopping tablet replica
I20260924 11:47:28.377305 4206 raft_consensus.cc:2229] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:28.377306 4207 raft_consensus.cc:2229] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:28.377512 4207 raft_consensus.cc:2258] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:28.377648 4206 raft_consensus.cc:2258] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:28.377771 4208 tablet_replica.cc:330] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: stopping tablet replica
I20260924 11:47:28.377950 4208 raft_consensus.cc:2229] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:28.377915 3614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 11:47:28.378152 4208 raft_consensus.cc:2258] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 11:47:28.378525 3614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:28.378793 4206 ts_tablet_manager.cc:1905] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:28.379179 4207 ts_tablet_manager.cc:1905] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:28.379191 4208 ts_tablet_manager.cc:1905] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:28.384037 4208 ts_tablet_manager.cc:1918] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:28.384136 4208 log.cc:1196] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/26f3ea950f1042898f673ffbd6e1eb0a
I20260924 11:47:28.384399 4208 ts_tablet_manager.cc:1939] T 26f3ea950f1042898f673ffbd6e1eb0a P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting consensus metadata
I20260924 11:47:28.385309 4206 ts_tablet_manager.cc:1918] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:28.385488 4206 log.cc:1196] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/26f3ea950f1042898f673ffbd6e1eb0a
I20260924 11:47:28.385694 4207 ts_tablet_manager.cc:1918] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:47:28.385722 3600 catalog_manager.cc:4841] TS ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253): tablet 26f3ea950f1042898f673ffbd6e1eb0a (table test-table [id=082e1956d0564aad8c13bb4bdc614014]) successfully deleted
I20260924 11:47:28.385818 4207 log.cc:1196] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/26f3ea950f1042898f673ffbd6e1eb0a
I20260924 11:47:28.385878 4206 ts_tablet_manager.cc:1939] T 26f3ea950f1042898f673ffbd6e1eb0a P 7ffd872059bf4404bc6927f8d4790f07: Deleting consensus metadata
I20260924 11:47:28.386355 4207 ts_tablet_manager.cc:1939] T 26f3ea950f1042898f673ffbd6e1eb0a P f8be1202d2fc4120a41b27fc188d96f1: Deleting consensus metadata
I20260924 11:47:28.386873 3599 catalog_manager.cc:4841] TS 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567): tablet 26f3ea950f1042898f673ffbd6e1eb0a (table test-table [id=082e1956d0564aad8c13bb4bdc614014]) successfully deleted
I20260924 11:47:28.387590 3602 catalog_manager.cc:4841] TS f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787): tablet 26f3ea950f1042898f673ffbd6e1eb0a (table test-table [id=082e1956d0564aad8c13bb4bdc614014]) successfully deleted
I20260924 11:47:28.389854 3751 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf3a93b70834271aa3df70b461a1574 (DEFAULT_TABLE table=test-table [id=821b60537b9c425ab67bf4168acf97eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:28.390174 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf3a93b70834271aa3df70b461a1574 (DEFAULT_TABLE table=test-table [id=821b60537b9c425ab67bf4168acf97eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:28.390277 3751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf3a93b70834271aa3df70b461a1574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:28.390259 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf3a93b70834271aa3df70b461a1574 (DEFAULT_TABLE table=test-table [id=821b60537b9c425ab67bf4168acf97eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:28.390547 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf3a93b70834271aa3df70b461a1574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:28.390586 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf3a93b70834271aa3df70b461a1574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:28.394817 4219 tablet_bootstrap.cc:492] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Bootstrap starting.
I20260924 11:47:28.396008 4219 tablet_bootstrap.cc:654] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:28.396553 4217 tablet_bootstrap.cc:492] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Bootstrap starting.
I20260924 11:47:28.398136 4217 tablet_bootstrap.cc:654] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:28.398746 4219 tablet_bootstrap.cc:492] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: No bootstrap required, opened a new log
I20260924 11:47:28.398907 4219 ts_tablet_manager.cc:1397] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:47:28.399199 4218 tablet_bootstrap.cc:492] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Bootstrap starting.
I20260924 11:47:28.399708 4219 raft_consensus.cc:348] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.399986 4217 tablet_bootstrap.cc:492] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: No bootstrap required, opened a new log
I20260924 11:47:28.400010 4219 raft_consensus.cc:374] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:28.400098 4219 raft_consensus.cc:729] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8be1202d2fc4120a41b27fc188d96f1, State: Initialized, Role: FOLLOWER
I20260924 11:47:28.400164 4217 ts_tablet_manager.cc:1397] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:47:28.400341 4219 consensus_queue.cc:260] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.400815 4218 tablet_bootstrap.cc:654] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:28.401196 4217 raft_consensus.cc:348] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.401489 4217 raft_consensus.cc:374] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:28.401628 4219 ts_tablet_manager.cc:1428] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 11:47:28.401633 4217 raft_consensus.cc:729] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Initialized, Role: FOLLOWER
I20260924 11:47:28.401870 4217 consensus_queue.cc:260] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.402374 4217 ts_tablet_manager.cc:1428] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:47:28.404747 4218 tablet_bootstrap.cc:492] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: No bootstrap required, opened a new log
I20260924 11:47:28.404964 4218 ts_tablet_manager.cc:1397] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 11:47:28.405812 4218 raft_consensus.cc:348] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.406095 4218 raft_consensus.cc:374] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:28.406190 4218 raft_consensus.cc:729] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdecbf8093944cc83fa9ce02ca2a83c, State: Initialized, Role: FOLLOWER
I20260924 11:47:28.406461 4218 consensus_queue.cc:260] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.406949 4218 ts_tablet_manager.cc:1428] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:47:28.518376 4201 raft_consensus.cc:482] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:28.518586 4201 raft_consensus.cc:504] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.519105 4201 leader_election.cc:288] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:28.519707 3773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf3a93b70834271aa3df70b461a1574" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1" is_pre_election: true
I20260924 11:47:28.519707 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf3a93b70834271aa3df70b461a1574" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" is_pre_election: true
I20260924 11:47:28.519963 3773 raft_consensus.cc:2454] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:28.519980 4050 raft_consensus.cc:2454] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 0.
I20260924 11:47:28.520864 3845 leader_election.cc:302] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:28.521245 4201 raft_consensus.cc:2781] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:28.521370 4201 raft_consensus.cc:482] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:28.521481 4201 raft_consensus.cc:3037] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:28.523543 4201 raft_consensus.cc:504] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.524155 4201 leader_election.cc:288] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Requested vote from peers f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787), ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253)
I20260924 11:47:28.524667 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf3a93b70834271aa3df70b461a1574" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c"
I20260924 11:47:28.524669 3773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf3a93b70834271aa3df70b461a1574" candidate_uuid: "7ffd872059bf4404bc6927f8d4790f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8be1202d2fc4120a41b27fc188d96f1"
I20260924 11:47:28.524888 3773 raft_consensus.cc:3037] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:28.524888 4050 raft_consensus.cc:3037] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:28.527781 4050 raft_consensus.cc:2454] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:28.527781 3773 raft_consensus.cc:2454] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ffd872059bf4404bc6927f8d4790f07 in term 1.
I20260924 11:47:28.528231 3845 leader_election.cc:302] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ffd872059bf4404bc6927f8d4790f07, ccdecbf8093944cc83fa9ce02ca2a83c, f8be1202d2fc4120a41b27fc188d96f1; no voters:
I20260924 11:47:28.528568 4201 raft_consensus.cc:2781] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:28.528748 4201 raft_consensus.cc:686] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Becoming Leader. State: Replica: 7ffd872059bf4404bc6927f8d4790f07, State: Running, Role: LEADER
I20260924 11:47:28.529007 4201 consensus_queue.cc:237] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } }
I20260924 11:47:28.531299 3614 catalog_manager.cc:5495] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "7ffd872059bf4404bc6927f8d4790f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ffd872059bf4404bc6927f8d4790f07" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 40567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:28.567028 3773 raft_consensus.cc:1261] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Refusing update from remote peer 7ffd872059bf4404bc6927f8d4790f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:28.567655 4184 consensus_queue.cc:1035] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8be1202d2fc4120a41b27fc188d96f1" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:28.567899 4049 raft_consensus.cc:1261] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Refusing update from remote peer 7ffd872059bf4404bc6927f8d4790f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:28.568560 4184 consensus_queue.cc:1035] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdecbf8093944cc83fa9ce02ca2a83c" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:47:28.587404 4189 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 11:47:30.165918 3615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:30.166131 3615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:30.170038 3615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e618ec21c32e41cb98a27b21adbb34b9: Sending DeleteTablet for 3 replicas of tablet 8cf3a93b70834271aa3df70b461a1574
I20260924 11:47:30.170825 3751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cf3a93b70834271aa3df70b461a1574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:30 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260924 11:47:30.170838 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cf3a93b70834271aa3df70b461a1574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:30 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:41116
I20260924 11:47:30.171121 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cf3a93b70834271aa3df70b461a1574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:30 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260924 11:47:30.171593 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3686
I20260924 11:47:30.180930 4259 tablet_replica.cc:330] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: stopping tablet replica
I20260924 11:47:30.181170 4258 tablet_replica.cc:330] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: stopping tablet replica
I20260924 11:47:30.181284 4259 raft_consensus.cc:2229] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:30.181519 4258 raft_consensus.cc:2229] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:30.181555 4259 raft_consensus.cc:2258] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:30.181759 4258 raft_consensus.cc:2258] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:30.182077 4257 tablet_replica.cc:330] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: stopping tablet replica
I20260924 11:47:30.182330 4257 raft_consensus.cc:2229] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:30.182614 4259 ts_tablet_manager.cc:1905] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:30.182614 4258 ts_tablet_manager.cc:1905] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:30.182780 4257 raft_consensus.cc:2258] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:30.185088 4257 ts_tablet_manager.cc:1905] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:30.189704 4259 ts_tablet_manager.cc:1918] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:47:30.189718 4258 ts_tablet_manager.cc:1918] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:47:30.189847 4258 log.cc:1196] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/8cf3a93b70834271aa3df70b461a1574
I20260924 11:47:30.189845 4259 log.cc:1196] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/8cf3a93b70834271aa3df70b461a1574
I20260924 11:47:30.190308 4259 ts_tablet_manager.cc:1939] T 8cf3a93b70834271aa3df70b461a1574 P ccdecbf8093944cc83fa9ce02ca2a83c: Deleting consensus metadata
I20260924 11:47:30.190333 4258 ts_tablet_manager.cc:1939] T 8cf3a93b70834271aa3df70b461a1574 P f8be1202d2fc4120a41b27fc188d96f1: Deleting consensus metadata
I20260924 11:47:30.190722 4257 ts_tablet_manager.cc:1918] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:47:30.190876 4257 log.cc:1196] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/8cf3a93b70834271aa3df70b461a1574
I20260924 11:47:30.191318 4257 ts_tablet_manager.cc:1939] T 8cf3a93b70834271aa3df70b461a1574 P 7ffd872059bf4404bc6927f8d4790f07: Deleting consensus metadata
I20260924 11:47:30.191457 3602 catalog_manager.cc:4841] TS f8be1202d2fc4120a41b27fc188d96f1 (127.1.210.1:39787): tablet 8cf3a93b70834271aa3df70b461a1574 (table test-table [id=821b60537b9c425ab67bf4168acf97eb]) successfully deleted
I20260924 11:47:30.191478 3600 catalog_manager.cc:4841] TS ccdecbf8093944cc83fa9ce02ca2a83c (127.1.210.3:36253): tablet 8cf3a93b70834271aa3df70b461a1574 (table test-table [id=821b60537b9c425ab67bf4168acf97eb]) successfully deleted
I20260924 11:47:30.192251 3599 catalog_manager.cc:4841] TS 7ffd872059bf4404bc6927f8d4790f07 (127.1.210.2:40567): tablet 8cf3a93b70834271aa3df70b461a1574 (table test-table [id=821b60537b9c425ab67bf4168acf97eb]) successfully deleted
Sep 24 11:47:30 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:30.202190 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:30.173291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.1.210.1:39787)
0924 11:47:30.179855 (+ 6564us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:30.179865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:30.180636 (+ 771us) server_negotiation.cc:407] Connection header received
0924 11:47:30.180861 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:30.180868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:30.180991 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:30.181301 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:30.183558 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:30.184302 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:30.186147 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:30.186882 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:30.187019 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:30.197121 (+ 10102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:30.197143 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:30.197161 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:30.197194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:30.199811 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:30.200400 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:30.200406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:30.200408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:30.200538 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:30.200837 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:30.200840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:30.200842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:30.201234 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:30.201449 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:30.201668 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 11:47:30.201825 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6371,"thread_start_us":6188,"threads_started":1}
W20260924 11:47:30.202802 3817 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54730
W20260924 11:47:30.203332 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3686: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:30.203514 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3686
I20260924 11:47:30.225661 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3826
Sep 24 11:47:30 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:30.239428 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:30.226355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.1.210.2:40567)
0924 11:47:30.226883 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:30.226888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:30.226903 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:47:30.226954 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:30.226958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:30.227021 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:30.227109 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:30.228244 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:30.228964 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:30.230204 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:30.230583 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:30.230675 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:30.234542 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:30.234554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:30.234569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:30.234598 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:30.237315 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:30.237876 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:30.237882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:30.237884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:30.238002 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:30.238287 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:30.238290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:30.238292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:30.238632 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:30.238836 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:30.239028 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 11:47:30.239177 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":305,"threads_started":1}
W20260924 11:47:30.240069 3956 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41136
W20260924 11:47:30.240644 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:30.240797 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3826
I20260924 11:47:30.261801 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3965
Sep 24 11:47:30 dist-test-slave-dd32 krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:30.275903 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:30.262505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.1.210.3:36253)
0924 11:47:30.263001 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:30.263005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:30.263021 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:47:30.263072 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:30.263075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:30.263130 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:30.263221 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:30.264383 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:30.265165 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:30.266342 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:30.266741 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:30.266828 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:30.270676 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:30.270689 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:30.270705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:30.270737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:30.273556 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:30.274187 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:30.274194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:30.274196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:30.274358 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:30.274683 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:30.274686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:30.274688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:30.275031 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:30.275251 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:30.275479 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 11:47:30.275631 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
W20260924 11:47:30.276503 4095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44660
W20260924 11:47:30.277143 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3965: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:30.277320 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3965
I20260924 11:47:30.294377 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 3583
I20260924 11:47:30.295471 3655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:30.408981 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 3583
2026-09-24T11:47:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8133 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:30 dist-test-slave-dd32 krb5kdc[4267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:30 dist-test-slave-dd32 krb5kdc[4267](info): set up 2 sockets
Sep 24 11:47:30 dist-test-slave-dd32 krb5kdc[4267](info): commencing operation
krb5kdc: starting...
W20260924 11:47:31.513244 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:31 dist-test-slave-dd32 krb5kdc[4267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:31.692462 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36023
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:36023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:32.010141 4280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:32.010576 4280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:32.010753 4280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:32.018350 4280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:32.018450 4280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:32.018496 4280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:32.018554 4280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:32.018596 4280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:32.027913 4280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:36023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:32.029244 4280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:32.030479 4280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:32.041699 4285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:32.041764 4288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:32.041795 4286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:32.042691 4280 server_base.cc:1034] running on GCE node
I20260924 11:47:32.043449 4280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:32.044764 4280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:32.045995 4280 hybrid_clock.cc:648] HybridClock initialized: now 1790250452045953 us; error 47 us; skew 500 ppm
I20260924 11:47:32.046386 4280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:32 dist-test-slave-dd32 krb5kdc[4267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:32.049479 4280 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:32.050930 4280 webserver.cc:460] Webserver started at http://127.1.210.62:43545/ using document root <none> and password file <none>
I20260924 11:47:32.051574 4280 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:32.051713 4280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:32.052014 4280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:32.055730 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "273944ac51684fa2a395b0fb701a3411"
format_stamp: "Formatted at 2026-09-24 11:47:32 on dist-test-slave-dd32"
server_key: "79a1a806b41d4f2160a2a425b1645453"
server_key_iv: "085bd45a9bcfb8d4d741ff6458db1786"
server_key_version: "encryptionkey@0"
I20260924 11:47:32.056625 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "273944ac51684fa2a395b0fb701a3411"
format_stamp: "Formatted at 2026-09-24 11:47:32 on dist-test-slave-dd32"
server_key: "79a1a806b41d4f2160a2a425b1645453"
server_key_iv: "085bd45a9bcfb8d4d741ff6458db1786"
server_key_version: "encryptionkey@0"
I20260924 11:47:32.061486 4280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260924 11:47:32.064749 4295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:32.066044 4280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:32.066251 4280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "273944ac51684fa2a395b0fb701a3411"
format_stamp: "Formatted at 2026-09-24 11:47:32 on dist-test-slave-dd32"
server_key: "79a1a806b41d4f2160a2a425b1645453"
server_key_iv: "085bd45a9bcfb8d4d741ff6458db1786"
server_key_version: "encryptionkey@0"
I20260924 11:47:32.066440 4280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:32.081290 4280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:32.085424 4280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:32.085693 4280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:32.103111 4280 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:36023
I20260924 11:47:32.103147 4346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:36023 every 8 connection(s)
I20260924 11:47:32.104378 4280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:32.107792 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:32.109573 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 4280
I20260924 11:47:32.109786 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:32.110350 1864 external_mini_cluster.cc:1428] Setting key 538b822c9e37650b4a888e0f9b4e7e79
I20260924 11:47:32.119656 4347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411: Bootstrap starting.
I20260924 11:47:32.123345 4347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:32.124629 4347 log.cc:824] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:47:32 dist-test-slave-dd32 krb5kdc[4267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:32.127317 4347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411: No bootstrap required, opened a new log
I20260924 11:47:32.133172 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:32.112269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.1.210.62:36023)
0924 11:47:32.113167 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:32.113178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:32.113216 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:47:32.115159 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:32.115192 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:32.115664 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:32.116085 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:32.117779 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:32.119070 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:32.120754 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:32.121480 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:32.121614 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:32.126228 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:32.126235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:32.126247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:32.126277 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:32.129362 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:32.130098 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:32.130106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:32.130112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:32.130269 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:32.130653 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:32.130656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:32.130658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:32.131129 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:32.131325 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:32.131768 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 11:47:32.132130 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":381,"threads_started":1}
I20260924 11:47:32.135703 4347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273944ac51684fa2a395b0fb701a3411" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36023 } }
I20260924 11:47:32.136082 4347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:32.136198 4347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273944ac51684fa2a395b0fb701a3411, State: Initialized, Role: FOLLOWER
I20260924 11:47:32.136718 4347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273944ac51684fa2a395b0fb701a3411" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36023 } }
I20260924 11:47:32.136938 4347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:32.137126 4347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:32.137286 4347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:32.139613 4347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273944ac51684fa2a395b0fb701a3411" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36023 } }
I20260924 11:47:32.140081 4347 leader_election.cc:302] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273944ac51684fa2a395b0fb701a3411; no voters:
I20260924 11:47:32.140619 4347 leader_election.cc:288] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:32.140713 4352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:32.141461 4352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [term 1 LEADER]: Becoming Leader. State: Replica: 273944ac51684fa2a395b0fb701a3411, State: Running, Role: LEADER
I20260924 11:47:32.141943 4352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273944ac51684fa2a395b0fb701a3411" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36023 } }
I20260924 11:47:32.142596 4347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:32.145413 4353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273944ac51684fa2a395b0fb701a3411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273944ac51684fa2a395b0fb701a3411" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36023 } } }
I20260924 11:47:32.145394 4354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273944ac51684fa2a395b0fb701a3411. Latest consensus state: current_term: 1 leader_uuid: "273944ac51684fa2a395b0fb701a3411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273944ac51684fa2a395b0fb701a3411" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36023 } } }
I20260924 11:47:32.145814 4353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:32.145846 4354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:32.148948 4359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:32.154012 4359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:32.162467 4359 catalog_manager.cc:1347] Generated new cluster ID: dee8b557096741c2b5a94a44433dac20
I20260924 11:47:32.162585 4359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:32.186347 4359 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:32.188167 4359 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:32.197037 4359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273944ac51684fa2a395b0fb701a3411: Generated new TSK 0
I20260924 11:47:32.198081 4359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:32.246805 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36023
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:32.569993 4373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:32.570304 4373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:32.570461 4373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:32.577594 4373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:32.577706 4373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:32.577876 4373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:32.587137 4373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:32.588340 4373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:32.589730 4373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:32.602527 4381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:32.602488 4379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:32.602717 4378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:32.603389 4373 server_base.cc:1034] running on GCE node
I20260924 11:47:32.603952 4373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:32.604817 4373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:32.605993 4373 hybrid_clock.cc:648] HybridClock initialized: now 1790250452605930 us; error 59 us; skew 500 ppm
I20260924 11:47:32.606279 4373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:32 dist-test-slave-dd32 krb5kdc[4267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:32.609136 4373 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:32.610432 4373 webserver.cc:460] Webserver started at http://127.1.210.1:40455/ using document root <none> and password file <none>
I20260924 11:47:32.611066 4373 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:32.611199 4373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:32.611496 4373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:32.614234 4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "53d9c7bf09a9422aa6d4fef0b4d02e5d"
format_stamp: "Formatted at 2026-09-24 11:47:32 on dist-test-slave-dd32"
server_key: "c25eb4e3487b147ffe157e021073ba69"
server_key_iv: "bd9136afbca94a034b9b5f3c3f354c09"
server_key_version: "encryptionkey@0"
I20260924 11:47:32.614971 4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "53d9c7bf09a9422aa6d4fef0b4d02e5d"
format_stamp: "Formatted at 2026-09-24 11:47:32 on dist-test-slave-dd32"
server_key: "c25eb4e3487b147ffe157e021073ba69"
server_key_iv: "bd9136afbca94a034b9b5f3c3f354c09"
server_key_version: "encryptionkey@0"
I20260924 11:47:32.619661 4373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 11:47:32.622761 4388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:32.623957 4373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:47:32.624118 4373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "53d9c7bf09a9422aa6d4fef0b4d02e5d"
format_stamp: "Formatted at 2026-09-24 11:47:32 on dist-test-slave-dd32"
server_key: "c25eb4e3487b147ffe157e021073ba69"
server_key_iv: "bd9136afbca94a034b9b5f3c3f354c09"
server_key_version: "encryptionkey@0"
I20260924 11:47:32.624305 4373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:32.636024 4373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:32.639586 4373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:32.639840 4373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:32.640663 4373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:32.641906 4373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:32.642004 4373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:32.642136 4373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:32.642203 4373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:32.673599 4373 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:38107
I20260924 11:47:32.673646 4500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:38107 every 8 connection(s)
I20260924 11:47:32.674827 4373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:32.684285 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 4373
I20260924 11:47:32.684487 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:32.685144 1864 external_mini_cluster.cc:1428] Setting key e8749ec962513e55d43f54283a599043
Sep 24 11:47:32 dist-test-slave-dd32 krb5kdc[4267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:32.693797 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:32.677193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:38727 (local address 127.1.210.62:36023)
0924 11:47:32.677611 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:32.677616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:32.678482 (+ 866us) server_negotiation.cc:407] Connection header received
0924 11:47:32.680274 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:32.680279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:32.680369 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:32.680480 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:32.681987 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:32.682684 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:32.684254 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:32.684758 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:32.684882 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:32.688424 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:32.688430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:32.688433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:32.688462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:32.690895 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:32.691612 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:32.691617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:32.691619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:32.691730 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:32.692137 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:32.692142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:32.692146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:32.692410 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:32.692570 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:32.693243 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 11:47:32.693452 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":223,"threads_started":1}
I20260924 11:47:32.694765 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:32.677674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36023 (local address 127.1.210.1:38727)
0924 11:47:32.678331 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:32.678373 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:32.680018 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:32.680706 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:32.680714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:32.681273 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:32.681827 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:32.681842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:32.682824 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:32.682829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:32.684115 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:32.684126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:32.684977 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:32.684983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:32.685296 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:32.686014 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:32.686035 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:32.688209 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:32.691031 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:32.691039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:32.691057 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:32.691462 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:32.691836 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:32.691841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:32.691845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:32.692021 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:32.692533 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:32.692551 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:32.692867 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 11:47:32.693154 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 11:47:32.693681 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
I20260924 11:47:32.696254 4501 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36023
I20260924 11:47:32.696555 4501 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:32.697427 4501 heartbeater.cc:511] Master 127.1.210.62:36023 requested a full tablet report, sending...
I20260924 11:47:32.699748 4312 ts_manager.cc:194] Registered new tserver with Master: 53d9c7bf09a9422aa6d4fef0b4d02e5d (127.1.210.1:38107)
I20260924 11:47:32.701897 4312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:38727
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:32.725575 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36023
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:33.036290 4507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:33.036547 4507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:33.036643 4507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:33.043772 4507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:33.043869 4507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:33.043989 4507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:33.053395 4507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:33.054513 4507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:33.055682 4507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:33.069796 4512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:33.069819 4515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:33.069756 4513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:33.070796 4507 server_base.cc:1034] running on GCE node
I20260924 11:47:33.071372 4507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:33.072242 4507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:33.073452 4507 hybrid_clock.cc:648] HybridClock initialized: now 1790250453073409 us; error 52 us; skew 500 ppm
I20260924 11:47:33.073750 4507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:33 dist-test-slave-dd32 krb5kdc[4267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:33.076638 4507 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:33.078016 4507 webserver.cc:460] Webserver started at http://127.1.210.2:44047/ using document root <none> and password file <none>
I20260924 11:47:33.078665 4507 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:33.078796 4507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:33.079092 4507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:33.081856 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "c3c93695a80d41ebbaeee7077d6fa8be"
format_stamp: "Formatted at 2026-09-24 11:47:33 on dist-test-slave-dd32"
server_key: "7c826c2aaef5a6e1c7bd25f518a66e54"
server_key_iv: "ff1b2fb8ccfef41611c5ca1d3a76cfd6"
server_key_version: "encryptionkey@0"
I20260924 11:47:33.082593 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "c3c93695a80d41ebbaeee7077d6fa8be"
format_stamp: "Formatted at 2026-09-24 11:47:33 on dist-test-slave-dd32"
server_key: "7c826c2aaef5a6e1c7bd25f518a66e54"
server_key_iv: "ff1b2fb8ccfef41611c5ca1d3a76cfd6"
server_key_version: "encryptionkey@0"
I20260924 11:47:33.087421 4507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:47:33.090622 4522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:33.091791 4507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:47:33.091946 4507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "c3c93695a80d41ebbaeee7077d6fa8be"
format_stamp: "Formatted at 2026-09-24 11:47:33 on dist-test-slave-dd32"
server_key: "7c826c2aaef5a6e1c7bd25f518a66e54"
server_key_iv: "ff1b2fb8ccfef41611c5ca1d3a76cfd6"
server_key_version: "encryptionkey@0"
I20260924 11:47:33.092144 4507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:33.119581 4507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:33.123301 4507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:33.123582 4507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:33.124444 4507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:33.125708 4507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:33.125806 4507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:33.125943 4507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:33.126011 4507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:33.157485 4507 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:36043
I20260924 11:47:33.157572 4634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:36043 every 8 connection(s)
I20260924 11:47:33.158720 4507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:33.161414 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 4507
I20260924 11:47:33.161597 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:33.161973 1864 external_mini_cluster.cc:1428] Setting key 56a8460084df8ccbed970fdf328c447e
Sep 24 11:47:33 dist-test-slave-dd32 krb5kdc[4267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:33.179265 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:33.161139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:46245 (local address 127.1.210.62:36023)
0924 11:47:33.161342 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:33.161348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:33.162645 (+ 1297us) server_negotiation.cc:407] Connection header received
0924 11:47:33.164463 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:33.164471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:33.164564 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:33.164671 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:33.166101 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.167288 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.168789 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.169261 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.169392 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:33.173395 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:33.173400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:33.173403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:33.173432 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:33.176165 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:33.176873 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:33.176881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:33.176887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:33.177085 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:33.177483 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:33.177487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:33.177492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:33.177849 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:33.178005 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:33.178640 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 11:47:33.178874 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:47:33.179940 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:33.161719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36023 (local address 127.1.210.2:46245)
0924 11:47:33.162502 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:33.162545 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:33.164217 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:33.164846 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:33.164853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:33.165407 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:33.165941 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:33.165957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.167445 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.167448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:33.168634 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:33.168646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.169470 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.169478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:33.169768 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:33.170448 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:33.170469 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:33.173202 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:33.176299 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:33.176305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:33.176317 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:33.176737 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:33.177194 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:33.177197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:33.177200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:33.177362 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:33.177964 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:33.177969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:33.178264 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 11:47:33.178476 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 11:47:33.178822 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
I20260924 11:47:33.181383 4635 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36023
I20260924 11:47:33.181674 4635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
I20260924 11:47:33.182502 4635 heartbeater.cc:511] Master 127.1.210.62:36023 requested a full tablet report, sending...
I20260924 11:47:33.183964 4312 ts_manager.cc:194] Registered new tserver with Master: c3c93695a80d41ebbaeee7077d6fa8be (127.1.210.2:36043)
I20260924 11:47:33.185210 4312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:46245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:33.204313 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36023
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:33.516036 4640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:33.516320 4640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:33.516422 4640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:33.524272 4640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:33.524428 4640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:33.524729 4640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:33.536774 4640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:33.538182 4640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:33.539492 4640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:33.553107 4645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:33.553398 4646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:33.554170 4648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:33.554769 4640 server_base.cc:1034] running on GCE node
I20260924 11:47:33.555365 4640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:33.556309 4640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:33.557642 4640 hybrid_clock.cc:648] HybridClock initialized: now 1790250453557579 us; error 56 us; skew 500 ppm
I20260924 11:47:33.557983 4640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:33 dist-test-slave-dd32 krb5kdc[4267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:33.561021 4640 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:33.562515 4640 webserver.cc:460] Webserver started at http://127.1.210.3:45605/ using document root <none> and password file <none>
I20260924 11:47:33.563189 4640 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:33.563323 4640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:33.563643 4640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:33.566512 4640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "21b3f55162a44ea68826b08264b2ccaf"
format_stamp: "Formatted at 2026-09-24 11:47:33 on dist-test-slave-dd32"
server_key: "35540733bf97ed3c2743812b9cf04874"
server_key_iv: "2717eff8354b7984d547a4d3695f3d3c"
server_key_version: "encryptionkey@0"
I20260924 11:47:33.567278 4640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "21b3f55162a44ea68826b08264b2ccaf"
format_stamp: "Formatted at 2026-09-24 11:47:33 on dist-test-slave-dd32"
server_key: "35540733bf97ed3c2743812b9cf04874"
server_key_iv: "2717eff8354b7984d547a4d3695f3d3c"
server_key_version: "encryptionkey@0"
I20260924 11:47:33.572446 4640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 11:47:33.576436 4655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:33.577766 4640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:47:33.577929 4640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "21b3f55162a44ea68826b08264b2ccaf"
format_stamp: "Formatted at 2026-09-24 11:47:33 on dist-test-slave-dd32"
server_key: "35540733bf97ed3c2743812b9cf04874"
server_key_iv: "2717eff8354b7984d547a4d3695f3d3c"
server_key_version: "encryptionkey@0"
I20260924 11:47:33.578115 4640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:33.589259 4640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:33.593472 4640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:33.593757 4640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:33.594602 4640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:33.595815 4640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:33.595885 4640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:33.596037 4640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:33.596105 4640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:33.628330 4640 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:34775
I20260924 11:47:33.628422 4767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:34775 every 8 connection(s)
I20260924 11:47:33.629678 4640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:33.630493 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 4640
I20260924 11:47:33.630682 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:33.631141 1864 external_mini_cluster.cc:1428] Setting key 1f7e2d1995bdc7160d69ab01b6da625e
Sep 24 11:47:33 dist-test-slave-dd32 krb5kdc[4267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:33.649276 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:33.632087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:36657 (local address 127.1.210.62:36023)
0924 11:47:33.632256 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:33.632260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:33.634432 (+ 2172us) server_negotiation.cc:407] Connection header received
0924 11:47:33.636116 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:33.636120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:33.636182 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:33.636264 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:33.637625 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.638351 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.640001 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.640386 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.640475 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:33.643670 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:33.643676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:33.643679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:33.643708 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:33.646408 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:33.647103 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:33.647112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:33.647115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:33.647220 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:33.647618 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:33.647623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:33.647627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:33.647916 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:33.648070 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:33.648700 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 11:47:33.648902 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:47:33.650199 4770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:33.633476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36023 (local address 127.1.210.3:36657)
0924 11:47:33.634292 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:33.634327 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:33.635874 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:33.636454 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:33.636461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:33.636925 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:33.637485 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:33.637501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.638484 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.638488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:33.639886 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:33.639898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:33.640561 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:33.640566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:33.640747 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:33.641402 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:33.641428 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:33.643463 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:33.646554 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:33.646563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:33.646578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:33.646975 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:33.647324 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:33.647329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:33.647333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:33.647505 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:33.648036 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:33.648044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:33.648354 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 11:47:33.648611 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 11:47:33.649113 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":365,"threads_started":1}
I20260924 11:47:33.651485 4768 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36023
I20260924 11:47:33.651793 4768 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:33.652623 4768 heartbeater.cc:511] Master 127.1.210.62:36023 requested a full tablet report, sending...
I20260924 11:47:33.654047 4312 ts_manager.cc:194] Registered new tserver with Master: 21b3f55162a44ea68826b08264b2ccaf (127.1.210.3:34775)
I20260924 11:47:33.654801 4312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:36657
I20260924 11:47:33.658290 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:47:33.688284 4503 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:33.686176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.1.210.62:36023)
0924 11:47:33.686379 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:33.686386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:33.686743 (+ 357us) server_negotiation.cc:407] Connection header received
0924 11:47:33.686889 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:33.686892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:33.686955 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:33.687048 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:33.688102 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45062: BlockingRecv error: recv got EOF from 127.0.0.1:45062 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:47:33.691349 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4373
W20260924 11:47:33.693976 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4373: Not authorized: Client connection negotiation failed: client connection to 127.1.210.1:38107: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:47:33.694134 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 4373
I20260924 11:47:33.710546 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4507
W20260924 11:47:33.713178 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4507: Not authorized: Client connection negotiation failed: client connection to 127.1.210.2:36043: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:47:33.713344 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 4507
I20260924 11:47:33.728490 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4640
W20260924 11:47:33.731261 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4640: Not authorized: Client connection negotiation failed: client connection to 127.1.210.3:34775: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:47:33.731397 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 4640
I20260924 11:47:33.746372 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4280
I20260924 11:47:33.747448 4342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:33.847473 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 4280
2026-09-24T11:47:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3436 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:34 dist-test-slave-dd32 krb5kdc[4785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:34 dist-test-slave-dd32 krb5kdc[4785](info): set up 2 sockets
Sep 24 11:47:34 dist-test-slave-dd32 krb5kdc[4785](info): commencing operation
krb5kdc: starting...
W20260924 11:47:35.257638 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.369s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:35 dist-test-slave-dd32 krb5kdc[4785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:35.433007 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:46465
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:46465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:35.760911 4798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:35.761276 4798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:35.761422 4798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:35.768406 4798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:35.768535 4798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:35.768633 4798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:35.768733 4798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:35.768816 4798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:35.778213 4798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:46465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:35.779417 4798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:35.780647 4798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:35.792757 4806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:35.793125 4803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:35.793125 4804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:35.793897 4798 server_base.cc:1034] running on GCE node
I20260924 11:47:35.794646 4798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:35.795987 4798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:35.797207 4798 hybrid_clock.cc:648] HybridClock initialized: now 1790250455797172 us; error 46 us; skew 500 ppm
I20260924 11:47:35.797537 4798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:35 dist-test-slave-dd32 krb5kdc[4785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:35.806156 4798 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:35.807516 4798 webserver.cc:460] Webserver started at http://127.1.210.62:38819/ using document root <none> and password file <none>
I20260924 11:47:35.808126 4798 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:35.808219 4798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:35.808478 4798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:35.811287 4798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "c0eb4aae8d25498ab948005e9046a2b5"
format_stamp: "Formatted at 2026-09-24 11:47:35 on dist-test-slave-dd32"
server_key: "0cef826ff467103aead8290b0405beb4"
server_key_iv: "18a0ad4e9a2e827c59640fbf6c06b7ab"
server_key_version: "encryptionkey@0"
I20260924 11:47:35.811975 4798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "c0eb4aae8d25498ab948005e9046a2b5"
format_stamp: "Formatted at 2026-09-24 11:47:35 on dist-test-slave-dd32"
server_key: "0cef826ff467103aead8290b0405beb4"
server_key_iv: "18a0ad4e9a2e827c59640fbf6c06b7ab"
server_key_version: "encryptionkey@0"
I20260924 11:47:35.816766 4798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 11:47:35.819922 4813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:35.821245 4798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:47:35.821377 4798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "c0eb4aae8d25498ab948005e9046a2b5"
format_stamp: "Formatted at 2026-09-24 11:47:35 on dist-test-slave-dd32"
server_key: "0cef826ff467103aead8290b0405beb4"
server_key_iv: "18a0ad4e9a2e827c59640fbf6c06b7ab"
server_key_version: "encryptionkey@0"
I20260924 11:47:35.821527 4798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:35.836304 4798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:35.839990 4798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:35.840209 4798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:35.857319 4798 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:46465
I20260924 11:47:35.857350 4864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:46465 every 8 connection(s)
I20260924 11:47:35.858556 4798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:35.859614 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 4798
I20260924 11:47:35.859817 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:35.860631 1864 external_mini_cluster.cc:1428] Setting key 26c5a845de4d3a10c0f203212e2f949e
I20260924 11:47:35.862993 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:47:35 dist-test-slave-dd32 krb5kdc[4785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:35.875526 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5: Bootstrap starting.
I20260924 11:47:35.878808 4865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:35.879828 4865 log.cc:824] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:35.882357 4868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:35.862397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.1.210.62:46465)
0924 11:47:35.863163 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:35.863174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:35.863212 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:47:35.864684 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:35.864704 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:35.865055 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:35.865469 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:35.867181 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:35.868480 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:35.870217 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:35.870842 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:35.870959 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:35.875397 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:35.875404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:35.875416 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:35.875451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:35.878311 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:35.878940 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:35.878953 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:35.878959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:35.879121 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:35.879468 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:35.879475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:35.879478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:35.879985 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:35.880286 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:35.880717 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 11:47:35.881147 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":358,"threads_started":1}
I20260924 11:47:35.882885 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5: No bootstrap required, opened a new log
I20260924 11:47:35.885839 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:47:35.890184 4865 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eb4aae8d25498ab948005e9046a2b5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46465 } }
I20260924 11:47:35.890504 4865 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:35.890640 4865 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0eb4aae8d25498ab948005e9046a2b5, State: Initialized, Role: FOLLOWER
I20260924 11:47:35.891207 4865 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eb4aae8d25498ab948005e9046a2b5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46465 } }
I20260924 11:47:35.891430 4865 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:35.891644 4865 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:35.891827 4865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:35.894232 4865 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eb4aae8d25498ab948005e9046a2b5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46465 } }
I20260924 11:47:35.894683 4865 leader_election.cc:302] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0eb4aae8d25498ab948005e9046a2b5; no voters:
I20260924 11:47:35.895198 4865 leader_election.cc:288] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:35.895318 4871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:35.897018 4865 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:35.901407 4871 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [term 1 LEADER]: Becoming Leader. State: Replica: c0eb4aae8d25498ab948005e9046a2b5, State: Running, Role: LEADER
I20260924 11:47:35.902086 4871 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eb4aae8d25498ab948005e9046a2b5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46465 } }
I20260924 11:47:35.905534 4872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0eb4aae8d25498ab948005e9046a2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eb4aae8d25498ab948005e9046a2b5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46465 } } }
I20260924 11:47:35.905858 4872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:35.906322 4873 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0eb4aae8d25498ab948005e9046a2b5. Latest consensus state: current_term: 1 leader_uuid: "c0eb4aae8d25498ab948005e9046a2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0eb4aae8d25498ab948005e9046a2b5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46465 } } }
I20260924 11:47:35.906812 4873 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5 [sys.catalog]: This master's current role is: LEADER
W20260924 11:47:35.912031 4885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:47:35.912151 4885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:47:35.912314 4886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:35.914757 4886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:35.923481 4886 catalog_manager.cc:1347] Generated new cluster ID: ecd11d00c7a24e98b568b35d4ab6e2e4
I20260924 11:47:35.923610 4886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:35.928140 4868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:35.919786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.1.210.62:46465)
0924 11:47:35.919957 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:35.919964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:35.920503 (+ 539us) server_negotiation.cc:407] Connection header received
0924 11:47:35.920776 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:35.920782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:35.920861 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:35.920973 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:35.922272 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:35.923443 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:35.924818 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:35.925390 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:35.925482 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:35.926715 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:35.926726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:35.926730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:35.927361 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:35.927414 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:35.927435 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:35.927552 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:35.927665 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 11:47:35.927754 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":63}
I20260924 11:47:35.972676 4886 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:35.974018 4886 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:35.980720 4886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0eb4aae8d25498ab948005e9046a2b5: Generated new TSK 0
I20260924 11:47:35.981523 4886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:47:35.991420 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4798
I20260924 11:47:35.992417 4860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:36.082397 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 4798
2026-09-24T11:47:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2228 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:36 dist-test-slave-dd32 krb5kdc[4899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:36 dist-test-slave-dd32 krb5kdc[4899](info): set up 2 sockets
Sep 24 11:47:36 dist-test-slave-dd32 krb5kdc[4899](info): commencing operation
krb5kdc: starting...
W20260924 11:47:37.197214 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:37 dist-test-slave-dd32 krb5kdc[4899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:37.381179 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43003
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:37.691915 4912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:37.692286 4912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:37.692453 4912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:37.699762 4912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:37.699931 4912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:37.700043 4912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:37.700142 4912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:37.700223 4912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:37.710350 4912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:37.711704 4912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:37.712996 4912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:37.724421 4917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:37.724377 4920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:37.724452 4918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:37.725322 4912 server_base.cc:1034] running on GCE node
I20260924 11:47:37.726122 4912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:37.727697 4912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:37.728935 4912 hybrid_clock.cc:648] HybridClock initialized: now 1790250457728877 us; error 43 us; skew 500 ppm
I20260924 11:47:37.729326 4912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:37 dist-test-slave-dd32 krb5kdc[4899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250457, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:37.732600 4912 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:37.734025 4912 webserver.cc:460] Webserver started at http://127.1.210.62:37313/ using document root <none> and password file <none>
I20260924 11:47:37.734714 4912 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:37.734822 4912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:37.735072 4912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:37.738211 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "c87b88ef68c64dee814582364eb5003d"
format_stamp: "Formatted at 2026-09-24 11:47:37 on dist-test-slave-dd32"
server_key: "10f8f8ae48a567617dfa43d14e17b9db"
server_key_iv: "8d46da6a24add3bb3788edf0ffbe039c"
server_key_version: "encryptionkey@0"
I20260924 11:47:37.738952 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "c87b88ef68c64dee814582364eb5003d"
format_stamp: "Formatted at 2026-09-24 11:47:37 on dist-test-slave-dd32"
server_key: "10f8f8ae48a567617dfa43d14e17b9db"
server_key_iv: "8d46da6a24add3bb3788edf0ffbe039c"
server_key_version: "encryptionkey@0"
I20260924 11:47:37.744086 4912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:47:37.747490 4927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:37.748847 4912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:37.748996 4912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "c87b88ef68c64dee814582364eb5003d"
format_stamp: "Formatted at 2026-09-24 11:47:37 on dist-test-slave-dd32"
server_key: "10f8f8ae48a567617dfa43d14e17b9db"
server_key_iv: "8d46da6a24add3bb3788edf0ffbe039c"
server_key_version: "encryptionkey@0"
I20260924 11:47:37.749241 4912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:37.767040 4912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:37.771586 4912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:37.771826 4912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:37.790761 4912 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43003
I20260924 11:47:37.790778 4978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43003 every 8 connection(s)
I20260924 11:47:37.792228 4912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:37.795857 4979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:37.799297 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 4912
I20260924 11:47:37.799494 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:37.800253 1864 external_mini_cluster.cc:1428] Setting key 3ad2d284628f4d4b57d069fb643d93f1
I20260924 11:47:37.808889 4979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d: Bootstrap starting.
I20260924 11:47:37.812433 4979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:37.813761 4979 log.cc:824] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:47:37 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:37.816493 4979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d: No bootstrap required, opened a new log
I20260924 11:47:37.822552 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:37.802257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.1.210.62:43003)
0924 11:47:37.803153 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:37.803165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:37.803206 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:47:37.804790 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:37.804810 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:37.805259 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:37.805590 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:37.807228 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:37.808573 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:37.810272 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:37.810909 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:37.811030 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:37.815499 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:37.815506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:37.815520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:37.815552 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:37.818628 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:37.819349 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:37.819355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:37.819361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:37.819526 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:37.819889 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:37.819894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:37.819897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:37.820421 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:37.820767 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:37.821138 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 11:47:37.821568 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":346,"threads_started":1}
I20260924 11:47:37.824170 4979 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87b88ef68c64dee814582364eb5003d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43003 } }
I20260924 11:47:37.824505 4979 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:37.824633 4979 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b88ef68c64dee814582364eb5003d, State: Initialized, Role: FOLLOWER
I20260924 11:47:37.825248 4979 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87b88ef68c64dee814582364eb5003d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43003 } }
I20260924 11:47:37.825503 4979 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:37.825616 4979 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:37.825739 4979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:37.827888 4979 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87b88ef68c64dee814582364eb5003d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43003 } }
I20260924 11:47:37.828387 4979 leader_election.cc:302] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87b88ef68c64dee814582364eb5003d; no voters:
I20260924 11:47:37.828979 4979 leader_election.cc:288] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:37.829118 4984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:37.829780 4984 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [term 1 LEADER]: Becoming Leader. State: Replica: c87b88ef68c64dee814582364eb5003d, State: Running, Role: LEADER
I20260924 11:47:37.830263 4984 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87b88ef68c64dee814582364eb5003d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43003 } }
I20260924 11:47:37.830883 4979 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:37.832633 4986 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87b88ef68c64dee814582364eb5003d. Latest consensus state: current_term: 1 leader_uuid: "c87b88ef68c64dee814582364eb5003d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87b88ef68c64dee814582364eb5003d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43003 } } }
I20260924 11:47:37.832973 4986 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:37.834240 4990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:37.833739 4985 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87b88ef68c64dee814582364eb5003d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87b88ef68c64dee814582364eb5003d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43003 } } }
I20260924 11:47:37.835171 4985 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:37.838488 4990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:37.846832 4990 catalog_manager.cc:1347] Generated new cluster ID: 951cc6061ad34c8782caf4e8e3d824c4
I20260924 11:47:37.846927 4990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:37.859117 4990 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:37.860324 4990 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:37.865540 4990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87b88ef68c64dee814582364eb5003d: Generated new TSK 0
I20260924 11:47:37.866248 4990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:37.909116 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43003
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:38.241741 5005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:38.242010 5005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:38.242107 5005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:38.249334 5005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:38.249442 5005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:38.249576 5005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:38.259035 5005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:38.260213 5005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:38.261556 5005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:38.274315 5010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:38.274412 5013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:38.274390 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:38.275446 5005 server_base.cc:1034] running on GCE node
I20260924 11:47:38.276024 5005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:38.276891 5005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:38.278116 5005 hybrid_clock.cc:648] HybridClock initialized: now 1790250458278068 us; error 59 us; skew 500 ppm
I20260924 11:47:38.278426 5005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:38 dist-test-slave-dd32 krb5kdc[4899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:38.281486 5005 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:38.282801 5005 webserver.cc:460] Webserver started at http://127.1.210.1:44893/ using document root <none> and password file <none>
I20260924 11:47:38.283464 5005 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:38.283610 5005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:38.283890 5005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:38.286634 5005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "5084e2219a83454db32a8852e51b4cbb"
format_stamp: "Formatted at 2026-09-24 11:47:38 on dist-test-slave-dd32"
server_key: "57692989d335fa0e4308cf7271b62c9d"
server_key_iv: "730652a7b871ae99fb2646d69b3a0183"
server_key_version: "encryptionkey@0"
I20260924 11:47:38.287395 5005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "5084e2219a83454db32a8852e51b4cbb"
format_stamp: "Formatted at 2026-09-24 11:47:38 on dist-test-slave-dd32"
server_key: "57692989d335fa0e4308cf7271b62c9d"
server_key_iv: "730652a7b871ae99fb2646d69b3a0183"
server_key_version: "encryptionkey@0"
I20260924 11:47:38.292076 5005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:47:38.295295 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:38.296520 5005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:47:38.296684 5005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "5084e2219a83454db32a8852e51b4cbb"
format_stamp: "Formatted at 2026-09-24 11:47:38 on dist-test-slave-dd32"
server_key: "57692989d335fa0e4308cf7271b62c9d"
server_key_iv: "730652a7b871ae99fb2646d69b3a0183"
server_key_version: "encryptionkey@0"
I20260924 11:47:38.296866 5005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:38.313269 5005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:38.317129 5005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:38.317399 5005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:38.318249 5005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:38.319550 5005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:38.319648 5005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:38.319787 5005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:38.319854 5005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:38.351781 5005 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:35539
I20260924 11:47:38.351815 5132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:35539 every 8 connection(s)
I20260924 11:47:38.353163 5005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:38.356264 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5005
I20260924 11:47:38.356462 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:38.357326 1864 external_mini_cluster.cc:1428] Setting key 7d4303a3f91fd0246922e5585b9c06b7
Sep 24 11:47:38 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:38.375062 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:38.355667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:49127 (local address 127.1.210.62:43003)
0924 11:47:38.356164 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:38.356171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:38.357319 (+ 1148us) server_negotiation.cc:407] Connection header received
0924 11:47:38.359440 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:38.359443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:38.359500 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:38.359580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:38.361674 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.362384 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.364812 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.365383 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.365517 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:38.369361 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:38.369365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:38.369369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:38.369397 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:38.371773 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:38.372672 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:38.372676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:38.372678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:38.372788 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:38.373292 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:38.373299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:38.373305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:38.373688 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:38.373859 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:38.374428 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 11:47:38.374653 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260924 11:47:38.375667 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:38.356229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43003 (local address 127.1.210.1:49127)
0924 11:47:38.357129 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:38.357198 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:38.359157 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:38.359845 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:38.359859 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:38.360616 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:38.361499 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:38.361526 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.362541 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.362549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:38.364649 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:38.364666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.365626 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.365644 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:38.365860 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:38.366539 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:38.366561 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:38.369162 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:38.371904 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:38.371916 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:38.371934 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:38.372554 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:38.372923 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:38.372928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:38.372932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:38.373170 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:38.373809 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:38.373814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:38.374083 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 11:47:38.374301 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 11:47:38.374641 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":325,"threads_started":1}
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:47:38.377103 5133 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43003
I20260924 11:47:38.377351 5133 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:38.377996 5133 heartbeater.cc:511] Master 127.1.210.62:43003 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:38.379801 4944 ts_manager.cc:194] Registered new tserver with Master: 5084e2219a83454db32a8852e51b4cbb (127.1.210.1:35539)
I20260924 11:47:38.381251 4944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:49127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:38.400157 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43003
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:38.711560 5139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:38.711822 5139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:38.711912 5139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:38.718880 5139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:38.718977 5139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:38.719085 5139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:38.728325 5139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:38.729573 5139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:38.730835 5139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:38.743347 5144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:38.743350 5145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:38.743607 5147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:38.744038 5139 server_base.cc:1034] running on GCE node
I20260924 11:47:38.744608 5139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:38.745561 5139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:38.746771 5139 hybrid_clock.cc:648] HybridClock initialized: now 1790250458746724 us; error 56 us; skew 500 ppm
I20260924 11:47:38.747056 5139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:38 dist-test-slave-dd32 krb5kdc[4899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:38.750072 5139 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:38.751376 5139 webserver.cc:460] Webserver started at http://127.1.210.2:41395/ using document root <none> and password file <none>
I20260924 11:47:38.752002 5139 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:38.752137 5139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:38.752434 5139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:38.755213 5139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "f1dc10eb482c4a5699fc435d66922c05"
format_stamp: "Formatted at 2026-09-24 11:47:38 on dist-test-slave-dd32"
server_key: "d5e69c57b98b3bc9e0c7c96a19e14dba"
server_key_iv: "f722d4df9c75b64e327fb65e5cd54cd8"
server_key_version: "encryptionkey@0"
I20260924 11:47:38.755930 5139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "f1dc10eb482c4a5699fc435d66922c05"
format_stamp: "Formatted at 2026-09-24 11:47:38 on dist-test-slave-dd32"
server_key: "d5e69c57b98b3bc9e0c7c96a19e14dba"
server_key_iv: "f722d4df9c75b64e327fb65e5cd54cd8"
server_key_version: "encryptionkey@0"
I20260924 11:47:38.760584 5139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:47:38.764273 5154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:38.765710 5139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:38.765875 5139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "f1dc10eb482c4a5699fc435d66922c05"
format_stamp: "Formatted at 2026-09-24 11:47:38 on dist-test-slave-dd32"
server_key: "d5e69c57b98b3bc9e0c7c96a19e14dba"
server_key_iv: "f722d4df9c75b64e327fb65e5cd54cd8"
server_key_version: "encryptionkey@0"
I20260924 11:47:38.766053 5139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:38.779035 5139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:38.782624 5139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:38.782934 5139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:38.783834 5139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:38.785141 5139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:38.785240 5139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:38.785382 5139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:38.785455 5139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:38.818532 5139 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:32823
I20260924 11:47:38.818571 5266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:32823 every 8 connection(s)
I20260924 11:47:38.819748 5139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:38.825484 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5139
I20260924 11:47:38.825662 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:38.826273 1864 external_mini_cluster.cc:1428] Setting key ffccb67d93a111e3caede34033cb6790
Sep 24 11:47:38 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:38.842458 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:38.822386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:42959 (local address 127.1.210.62:43003)
0924 11:47:38.822560 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:38.822564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:38.823705 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 11:47:38.825663 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:38.825667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:38.825761 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:38.825880 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:38.827594 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.828431 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.830937 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.831332 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.831435 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:38.835676 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:38.835685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:38.835692 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:38.835731 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:38.839150 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:38.839882 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:38.839890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:38.839896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:38.840046 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:38.840569 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:38.840576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:38.840582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:38.840973 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:38.841165 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:38.841806 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 11:47:38.842056 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 11:47:38.843111 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:38.822896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43003 (local address 127.1.210.2:42959)
0924 11:47:38.823553 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:38.823589 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:38.825348 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:38.826128 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:38.826151 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:38.826821 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:38.827445 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:38.827467 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.828574 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.828581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:38.830813 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:38.830828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:38.831545 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:38.831555 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:38.831880 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:38.832546 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:38.832577 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:38.835485 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:38.839279 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:38.839285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:38.839296 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:38.839762 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:38.840164 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:38.840170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:38.840175 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:38.840422 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:38.841123 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:38.841130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:38.841439 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 11:47:38.841655 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 11:47:38.842078 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":262,"threads_started":1}
I20260924 11:47:38.844400 5267 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43003
I20260924 11:47:38.844725 5267 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:38.845677 5267 heartbeater.cc:511] Master 127.1.210.62:43003 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:47:38.847222 4944 ts_manager.cc:194] Registered new tserver with Master: f1dc10eb482c4a5699fc435d66922c05 (127.1.210.2:32823)
I20260924 11:47:38.847990 4944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:42959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:38.870193 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43003
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:39.182883 5272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:39.183146 5272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:39.183347 5272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:39.190476 5272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:39.190572 5272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:39.190743 5272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:39.200212 5272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:39.201475 5272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:39.202723 5272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:39.215574 5277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:39.215567 5278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:39.215619 5280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:39.215969 5272 server_base.cc:1034] running on GCE node
I20260924 11:47:39.216749 5272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:39.217712 5272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:39.218930 5272 hybrid_clock.cc:648] HybridClock initialized: now 1790250459218883 us; error 37 us; skew 500 ppm
I20260924 11:47:39.219223 5272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:39.222170 5272 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:39.223484 5272 webserver.cc:460] Webserver started at http://127.1.210.3:35247/ using document root <none> and password file <none>
I20260924 11:47:39.224156 5272 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:39.224298 5272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:39.224603 5272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:39.227411 5272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "61853d28ebcc456aa12689265cf00594"
format_stamp: "Formatted at 2026-09-24 11:47:39 on dist-test-slave-dd32"
server_key: "09aebe3c010437e83c3de9a5311cc448"
server_key_iv: "de4de772a8fe6ed152dcfd28a10f24c1"
server_key_version: "encryptionkey@0"
I20260924 11:47:39.228153 5272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "61853d28ebcc456aa12689265cf00594"
format_stamp: "Formatted at 2026-09-24 11:47:39 on dist-test-slave-dd32"
server_key: "09aebe3c010437e83c3de9a5311cc448"
server_key_iv: "de4de772a8fe6ed152dcfd28a10f24c1"
server_key_version: "encryptionkey@0"
I20260924 11:47:39.233009 5272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 11:47:39.236228 5287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:39.237497 5272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:47:39.237663 5272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "61853d28ebcc456aa12689265cf00594"
format_stamp: "Formatted at 2026-09-24 11:47:39 on dist-test-slave-dd32"
server_key: "09aebe3c010437e83c3de9a5311cc448"
server_key_iv: "de4de772a8fe6ed152dcfd28a10f24c1"
server_key_version: "encryptionkey@0"
I20260924 11:47:39.237861 5272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:39.250348 5272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:39.254467 5272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:39.254745 5272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:39.255590 5272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:39.256783 5272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:39.256851 5272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:39.257007 5272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:39.257095 5272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:39.286600 5272 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:39559
I20260924 11:47:39.286643 5399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:39559 every 8 connection(s)
I20260924 11:47:39.287812 5272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:39.295928 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5272
I20260924 11:47:39.296078 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:39.296485 1864 external_mini_cluster.cc:1428] Setting key 238494162b2e1dc21617c38f1b36ee62
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:39.306579 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:39.290234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:50727 (local address 127.1.210.62:43003)
0924 11:47:39.290424 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:39.290430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:39.291556 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 11:47:39.293301 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:39.293305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:39.293365 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:39.293442 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:39.294795 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.295490 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.297084 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.297654 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.297783 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:39.301166 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:39.301172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:39.301182 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:39.301210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:39.303792 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.304438 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.304445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.304449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.304569 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.304948 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.304954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.304957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.305279 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:39.305425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:39.306036 (+ 611us) server_negotiation.cc:299] Negotiation successful
0924 11:47:39.306247 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:47:39.307427 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:39.290762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43003 (local address 127.1.210.3:50727)
0924 11:47:39.291417 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:39.291451 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:39.292999 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:39.293655 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:39.293662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:39.294130 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:39.294668 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:39.294683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.295621 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.295625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:39.296908 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:39.296919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.297856 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.297866 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:39.298142 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:39.298856 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:39.298886 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:39.300949 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:39.303927 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:39.303935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:39.303949 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:39.304325 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:39.304669 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:39.304675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:39.304679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:39.304847 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:39.305395 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:39.305402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:39.305718 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 11:47:39.305954 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 11:47:39.306421 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":261,"threads_started":1}
I20260924 11:47:39.308745 5400 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43003
I20260924 11:47:39.309026 5400 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:39.309753 5400 heartbeater.cc:511] Master 127.1.210.62:43003 requested a full tablet report, sending...
I20260924 11:47:39.311255 4943 ts_manager.cc:194] Registered new tserver with Master: 61853d28ebcc456aa12689265cf00594 (127.1.210.3:39559)
I20260924 11:47:39.312073 4943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:50727
I20260924 11:47:39.323093 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:39.367982 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:39.355217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.1.210.62:43003)
0924 11:47:39.355406 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:39.355413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:39.355955 (+ 542us) server_negotiation.cc:407] Connection header received
0924 11:47:39.356096 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:39.356102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:39.356180 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:39.356283 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:39.357697 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.358410 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.359533 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.359914 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.360002 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:39.363491 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:39.363496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:39.363499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:39.363529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:39.366007 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.366548 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.366551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.366554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.366671 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.366966 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.366968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.366970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.367242 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:39.367412 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:39.367593 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 11:47:39.367769 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 11:47:39.368466 4943 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59314
I20260924 11:47:39.370306 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5005
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:39.383960 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:39.371199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.1.210.1:35539)
0924 11:47:39.371741 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:39.371748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:39.371772 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:47:39.371820 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:39.371827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:39.372016 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:39.372129 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:39.373167 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.374107 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.375181 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.375708 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.375802 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:39.379194 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:39.379204 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:39.379216 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:39.379243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:39.381894 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.382403 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.382410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.382412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.382523 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.382808 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.382811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.382812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.383095 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:39.383311 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:39.383467 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 11:47:39.383637 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
I20260924 11:47:39.384074 5133 heartbeater.cc:503] Master 127.1.210.62:43003 was elected leader, sending a full tablet report...
W20260924 11:47:39.385251 5128 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37474
W20260924 11:47:39.386528 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5005: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:39.386730 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5005
I20260924 11:47:39.402976 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5139
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:39.417917 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:39.403724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.1.210.2:32823)
0924 11:47:39.404237 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:39.404244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:39.404267 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:47:39.404320 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:39.404326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:39.404532 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:39.404686 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:39.405988 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.406960 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.408209 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.408744 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.408879 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:39.412750 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:39.412759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:39.412771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:39.412801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:39.415327 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.415960 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.415967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.415969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.416076 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.416430 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.416437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.416440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.416753 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:39.417104 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:39.417341 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 11:47:39.417591 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
W20260924 11:47:39.419157 5262 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39424
W20260924 11:47:39.420184 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5139: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:39.420300 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5139
I20260924 11:47:39.435390 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5272
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[4899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250459, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:39.450470 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:39.436237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.1.210.3:39559)
0924 11:47:39.436807 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:39.436813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:39.436840 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:47:39.436890 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:39.436895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:39.437136 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:39.437308 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:39.438582 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.439679 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.440925 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:39.441530 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:39.441660 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:39.445647 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:39.445659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:39.445672 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:39.445703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:39.448136 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.448724 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.448730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.448733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.448840 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:39.449229 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:39.449236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:39.449242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:39.449550 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:39.449776 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:39.450022 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 11:47:39.450208 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":303,"threads_started":1}
W20260924 11:47:39.451717 5395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51874
W20260924 11:47:39.452747 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:47:39.452900 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5272
I20260924 11:47:39.467985 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 4912
I20260924 11:47:39.469136 4974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:39.569355 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 4912
2026-09-24T11:47:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3492 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[5417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[5417](info): set up 2 sockets
Sep 24 11:47:39 dist-test-slave-dd32 krb5kdc[5417](info): commencing operation
krb5kdc: starting...
W20260924 11:47:40.645480 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:40 dist-test-slave-dd32 krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:40.827975 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:38691
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:38691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:41.140753 5430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:41.141105 5430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:41.141291 5430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:41.148334 5430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:41.148494 5430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:41.148605 5430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:41.148705 5430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:41.148798 5430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:41.158484 5430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:38691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:38691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:41.159847 5430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:41.161156 5430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:41.172688 5438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:41.172796 5436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:41.172791 5435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:41.173756 5430 server_base.cc:1034] running on GCE node
I20260924 11:47:41.174518 5430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:41.175796 5430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:41.177021 5430 hybrid_clock.cc:648] HybridClock initialized: now 1790250461176980 us; error 49 us; skew 500 ppm
I20260924 11:47:41.177388 5430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:41 dist-test-slave-dd32 krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:41.183259 5430 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:41.184674 5430 webserver.cc:460] Webserver started at http://127.1.210.62:46415/ using document root <none> and password file <none>
I20260924 11:47:41.185410 5430 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:41.185545 5430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:41.185851 5430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:41.188643 5430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "9092f8f6c33d420d82f16b5fed629b45"
format_stamp: "Formatted at 2026-09-24 11:47:41 on dist-test-slave-dd32"
server_key: "a54fe93c4e080c51d3af2966554e2c53"
server_key_iv: "e5f8701c0d9aeb621c71d858a4bbb538"
server_key_version: "encryptionkey@0"
I20260924 11:47:41.189445 5430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "9092f8f6c33d420d82f16b5fed629b45"
format_stamp: "Formatted at 2026-09-24 11:47:41 on dist-test-slave-dd32"
server_key: "a54fe93c4e080c51d3af2966554e2c53"
server_key_iv: "e5f8701c0d9aeb621c71d858a4bbb538"
server_key_version: "encryptionkey@0"
I20260924 11:47:41.194332 5430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:47:41.197608 5445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:41.198836 5430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:47:41.199008 5430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "9092f8f6c33d420d82f16b5fed629b45"
format_stamp: "Formatted at 2026-09-24 11:47:41 on dist-test-slave-dd32"
server_key: "a54fe93c4e080c51d3af2966554e2c53"
server_key_iv: "e5f8701c0d9aeb621c71d858a4bbb538"
server_key_version: "encryptionkey@0"
I20260924 11:47:41.199218 5430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:41.232535 5430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:41.236346 5430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:41.236618 5430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:41.256044 5430 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:38691
I20260924 11:47:41.256096 5496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:38691 every 8 connection(s)
I20260924 11:47:41.257490 5430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:41.261184 5497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:41.262342 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5430
I20260924 11:47:41.262559 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:41.263132 1864 external_mini_cluster.cc:1428] Setting key 8f65c3166422267bf985034c7f640679
I20260924 11:47:41.274891 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45: Bootstrap starting.
Sep 24 11:47:41 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:41.277733 5497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:41.279021 5497 log.cc:824] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:41.282128 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45: No bootstrap required, opened a new log
I20260924 11:47:41.285748 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:41.265036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.1.210.62:38691)
0924 11:47:41.265895 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:41.265910 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:41.265957 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:47:41.267574 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:41.267599 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:41.268099 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:41.268518 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:41.269822 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:41.271791 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:41.272941 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:41.273978 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:41.274141 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:41.277856 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:41.277866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:41.277884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:41.277927 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:41.281640 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:41.282222 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:41.282229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:41.282237 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:41.282425 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:41.282768 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:41.282773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:41.282776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:41.283404 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:41.283708 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:41.284020 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 11:47:41.284432 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":330,"threads_started":1}
I20260924 11:47:41.291219 5497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9092f8f6c33d420d82f16b5fed629b45" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38691 } }
I20260924 11:47:41.291524 5497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:41.291597 5497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9092f8f6c33d420d82f16b5fed629b45, State: Initialized, Role: FOLLOWER
I20260924 11:47:41.292086 5497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9092f8f6c33d420d82f16b5fed629b45" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38691 } }
I20260924 11:47:41.292290 5497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:41.292382 5497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:41.292496 5497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:41.294458 5497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9092f8f6c33d420d82f16b5fed629b45" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38691 } }
I20260924 11:47:41.294838 5497 leader_election.cc:302] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9092f8f6c33d420d82f16b5fed629b45; no voters:
I20260924 11:47:41.295297 5497 leader_election.cc:288] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:41.295425 5502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:41.296072 5502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [term 1 LEADER]: Becoming Leader. State: Replica: 9092f8f6c33d420d82f16b5fed629b45, State: Running, Role: LEADER
I20260924 11:47:41.296573 5502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9092f8f6c33d420d82f16b5fed629b45" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38691 } }
I20260924 11:47:41.297354 5497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:41.299602 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9092f8f6c33d420d82f16b5fed629b45. Latest consensus state: current_term: 1 leader_uuid: "9092f8f6c33d420d82f16b5fed629b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9092f8f6c33d420d82f16b5fed629b45" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38691 } } }
I20260924 11:47:41.300261 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:41.299620 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9092f8f6c33d420d82f16b5fed629b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9092f8f6c33d420d82f16b5fed629b45" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38691 } } }
I20260924 11:47:41.301050 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:41.302122 5511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:41.306452 5511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:41.314388 5511 catalog_manager.cc:1347] Generated new cluster ID: f73a8d6d85804b4abb37a6f552905f02
I20260924 11:47:41.314505 5511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:41.336570 5511 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:41.338348 5511 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:41.345430 5511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9092f8f6c33d420d82f16b5fed629b45: Generated new TSK 0
I20260924 11:47:41.346387 5511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:41.389961 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:38691
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:41.700801 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:41.701140 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:41.701339 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:41.708851 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:41.708984 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:41.709172 5523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:41.718907 5523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:38691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:41.720034 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:41.721319 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:41.733917 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:41.734038 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:41.734053 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:41.735450 5523 server_base.cc:1034] running on GCE node
I20260924 11:47:41.736042 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:41.736920 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:41.738154 5523 hybrid_clock.cc:648] HybridClock initialized: now 1790250461738094 us; error 55 us; skew 500 ppm
I20260924 11:47:41.738447 5523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:41 dist-test-slave-dd32 krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:41.741499 5523 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:41.742749 5523 webserver.cc:460] Webserver started at http://127.1.210.1:45979/ using document root <none> and password file <none>
I20260924 11:47:41.743451 5523 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:41.743593 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:41.743878 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:41.746655 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "f8c93c4fd3a54bf9974508e5f4b866c2"
format_stamp: "Formatted at 2026-09-24 11:47:41 on dist-test-slave-dd32"
server_key: "c777d6047f7f59472a61424fb4211831"
server_key_iv: "c67da688b56fcb7230823bba5e0559e1"
server_key_version: "encryptionkey@0"
I20260924 11:47:41.747398 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "f8c93c4fd3a54bf9974508e5f4b866c2"
format_stamp: "Formatted at 2026-09-24 11:47:41 on dist-test-slave-dd32"
server_key: "c777d6047f7f59472a61424fb4211831"
server_key_iv: "c67da688b56fcb7230823bba5e0559e1"
server_key_version: "encryptionkey@0"
I20260924 11:47:41.752166 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:47:41.755394 5538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:41.756608 5523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:47:41.756762 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "f8c93c4fd3a54bf9974508e5f4b866c2"
format_stamp: "Formatted at 2026-09-24 11:47:41 on dist-test-slave-dd32"
server_key: "c777d6047f7f59472a61424fb4211831"
server_key_iv: "c67da688b56fcb7230823bba5e0559e1"
server_key_version: "encryptionkey@0"
I20260924 11:47:41.756945 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:41.775934 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:41.780102 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:41.780398 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:41.781342 5523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:41.782624 5523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:41.782729 5523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:41.782866 5523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:41.782934 5523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:41.814615 5523 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:35635
I20260924 11:47:41.814661 5650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:35635 every 8 connection(s)
I20260924 11:47:41.815878 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:41.817760 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5523
I20260924 11:47:41.817970 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:41.818643 1864 external_mini_cluster.cc:1428] Setting key ed5dfc2e5555736d004b68659e0b321b
Sep 24 11:47:41 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:47:41.836397 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:41.818789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:57199 (local address 127.1.210.62:38691)
0924 11:47:41.819238 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:41.819244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:41.820384 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 11:47:41.822308 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:41.822314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:41.822392 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:41.822513 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:41.824027 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:41.824760 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:41.826545 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:41.826938 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:41.827038 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:41.831046 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:41.831057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:41.831061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:41.831090 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:41.833593 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:41.834272 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:41.834277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:41.834285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:41.834388 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:41.834796 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:41.834801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:41.834804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:41.835060 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:41.835216 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:41.835832 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 11:47:41.836032 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
I20260924 11:47:41.837396 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:41.819376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:38691 (local address 127.1.210.1:57199)
0924 11:47:41.820223 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:41.820270 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:41.822036 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:41.822780 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:41.822789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:41.823309 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:41.823865 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:41.823882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:41.824906 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:41.824910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:41.826405 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:41.826422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:41.827156 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:41.827165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:41.827496 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:41.828197 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:41.828247 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:41.830843 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:41.833733 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:41.833741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:41.833756 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:41.834146 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:41.834490 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:41.834496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:41.834500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:41.834676 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:41.835179 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:41.835187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:41.835491 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 11:47:41.835752 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 11:47:41.836238 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:41.838891 5651 heartbeater.cc:347] Connected to a master server at 127.1.210.62:38691
I20260924 11:47:41.839174 5651 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:41.839855 5651 heartbeater.cc:511] Master 127.1.210.62:38691 requested a full tablet report, sending...
I20260924 11:47:41.842144 5462 ts_manager.cc:194] Registered new tserver with Master: f8c93c4fd3a54bf9974508e5f4b866c2 (127.1.210.1:35635)
I20260924 11:47:41.843755 5462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:57199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:41.860179 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:38691
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:42.172374 5657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:42.172639 5657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:42.172729 5657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:42.179932 5657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:42.180042 5657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:42.180167 5657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:42.189765 5657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:38691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:42.191025 5657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:42.192245 5657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:42.205235 5662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:42.205231 5663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:42.205510 5665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:42.206037 5657 server_base.cc:1034] running on GCE node
I20260924 11:47:42.206600 5657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:42.207494 5657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:42.208719 5657 hybrid_clock.cc:648] HybridClock initialized: now 1790250462208667 us; error 62 us; skew 500 ppm
I20260924 11:47:42.209015 5657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:42.212056 5657 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:42.213558 5657 webserver.cc:460] Webserver started at http://127.1.210.2:44681/ using document root <none> and password file <none>
I20260924 11:47:42.214205 5657 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:42.214303 5657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:42.214545 5657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:42.217444 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "90d0de3686604a11aac1376bd0c197da"
format_stamp: "Formatted at 2026-09-24 11:47:42 on dist-test-slave-dd32"
server_key: "47c0b4b31e713ee25d50b1e78cb3c675"
server_key_iv: "8a3ea0a2ba7b970c85e21196c06c92cd"
server_key_version: "encryptionkey@0"
I20260924 11:47:42.218139 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "90d0de3686604a11aac1376bd0c197da"
format_stamp: "Formatted at 2026-09-24 11:47:42 on dist-test-slave-dd32"
server_key: "47c0b4b31e713ee25d50b1e78cb3c675"
server_key_iv: "8a3ea0a2ba7b970c85e21196c06c92cd"
server_key_version: "encryptionkey@0"
I20260924 11:47:42.222856 5657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:47:42.226089 5672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:42.227344 5657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:47:42.227500 5657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "90d0de3686604a11aac1376bd0c197da"
format_stamp: "Formatted at 2026-09-24 11:47:42 on dist-test-slave-dd32"
server_key: "47c0b4b31e713ee25d50b1e78cb3c675"
server_key_iv: "8a3ea0a2ba7b970c85e21196c06c92cd"
server_key_version: "encryptionkey@0"
I20260924 11:47:42.227707 5657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:42.248302 5657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:42.251926 5657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:42.252187 5657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:42.253052 5657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:42.254359 5657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:42.254436 5657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:42.254590 5657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:42.254638 5657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:42.288300 5657 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:37547
I20260924 11:47:42.288371 5784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:37547 every 8 connection(s)
I20260924 11:47:42.289565 5657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:42.296047 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5657
I20260924 11:47:42.296247 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:42.296630 1864 external_mini_cluster.cc:1428] Setting key 6dea9e99345b14c8777a9bcda699ec5f
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:42.308533 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.291938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:34575 (local address 127.1.210.62:38691)
0924 11:47:42.292107 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:42.292111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:42.293227 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 11:47:42.294952 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:42.294957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:42.295045 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:42.295149 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:42.296630 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.297587 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.299126 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.299549 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.299671 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.303085 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:42.303090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:42.303094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:42.303123 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:42.305773 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.306443 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.306451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.306454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.306585 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.306985 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.306990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.306994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.307231 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:42.307375 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:42.307983 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 11:47:42.308190 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:47:42.309648 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.292350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:38691 (local address 127.1.210.2:34575)
0924 11:47:42.293008 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:42.293048 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:42.294699 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:42.295376 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:42.295385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:42.295869 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:42.296476 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:42.296497 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.297723 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.297732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:42.298995 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:42.299006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.299763 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.299771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:42.299985 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.300644 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:42.300665 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:42.302892 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:42.305904 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:42.305913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:42.305928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:42.306330 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:42.306689 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:42.306695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:42.306699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:42.306870 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:42.307349 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:42.307357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:42.307662 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 11:47:42.307911 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 11:47:42.308415 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":270,"threads_started":1}
I20260924 11:47:42.311182 5785 heartbeater.cc:347] Connected to a master server at 127.1.210.62:38691
I20260924 11:47:42.311483 5785 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:42.312155 5785 heartbeater.cc:511] Master 127.1.210.62:38691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:47:42.313650 5462 ts_manager.cc:194] Registered new tserver with Master: 90d0de3686604a11aac1376bd0c197da (127.1.210.2:37547)
I20260924 11:47:42.314513 5462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:34575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:42.340097 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:38691
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:42.659575 5790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:42.659893 5790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:42.660043 5790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:42.667263 5790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:42.667380 5790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:42.667573 5790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:42.676818 5790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:38691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:42.678063 5790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:42.679354 5790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:42.692116 5798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:42.692227 5795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:42.692193 5796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:42.693183 5790 server_base.cc:1034] running on GCE node
I20260924 11:47:42.693836 5790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:42.694764 5790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:42.696058 5790 hybrid_clock.cc:648] HybridClock initialized: now 1790250462696002 us; error 39 us; skew 500 ppm
I20260924 11:47:42.696400 5790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:42.699723 5790 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:42.701030 5790 webserver.cc:460] Webserver started at http://127.1.210.3:37935/ using document root <none> and password file <none>
I20260924 11:47:42.701747 5790 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:42.701846 5790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:42.702090 5790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:42.705169 5790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "65765790a8db4ac0b25f3d4f02600183"
format_stamp: "Formatted at 2026-09-24 11:47:42 on dist-test-slave-dd32"
server_key: "7d14a105a2893c44fbbbca4b725dfac5"
server_key_iv: "2f54ed4893798ecd5b8c7594e2e97052"
server_key_version: "encryptionkey@0"
I20260924 11:47:42.705881 5790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "65765790a8db4ac0b25f3d4f02600183"
format_stamp: "Formatted at 2026-09-24 11:47:42 on dist-test-slave-dd32"
server_key: "7d14a105a2893c44fbbbca4b725dfac5"
server_key_iv: "2f54ed4893798ecd5b8c7594e2e97052"
server_key_version: "encryptionkey@0"
I20260924 11:47:42.710992 5790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:47:42.714599 5805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:42.715869 5790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:47:42.715989 5790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "65765790a8db4ac0b25f3d4f02600183"
format_stamp: "Formatted at 2026-09-24 11:47:42 on dist-test-slave-dd32"
server_key: "7d14a105a2893c44fbbbca4b725dfac5"
server_key_iv: "2f54ed4893798ecd5b8c7594e2e97052"
server_key_version: "encryptionkey@0"
I20260924 11:47:42.716168 5790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:42.741578 5790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:42.745452 5790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:42.745724 5790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:42.746654 5790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:42.747838 5790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:42.747907 5790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:42.747984 5790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:42.748025 5790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:42.779770 5790 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:36759
I20260924 11:47:42.779855 5917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:36759 every 8 connection(s)
I20260924 11:47:42.781033 5790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:42.787560 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5790
I20260924 11:47:42.787766 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:42.788163 1864 external_mini_cluster.cc:1428] Setting key 573e8b2f88a3166ed191e0615877d0ef
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:42.800493 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.783508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:33423 (local address 127.1.210.62:38691)
0924 11:47:42.783680 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:42.783685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:42.784799 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 11:47:42.786612 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:42.786617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:42.786702 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:42.786805 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:42.788404 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.789401 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.790933 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.791480 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.791646 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.794954 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:42.794960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:42.794965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:42.795004 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:42.797566 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.798268 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.798274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.798278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.798445 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.798846 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.798852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.798856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.799120 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:42.799279 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:42.799927 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 11:47:42.800152 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:47:42.801489 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.783954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:38691 (local address 127.1.210.3:33423)
0924 11:47:42.784622 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:42.784664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:42.786356 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:42.787027 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:42.787036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:42.787541 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:42.788179 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:42.788197 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.789550 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.789555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:42.790790 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:42.790802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.791735 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.791743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:42.791970 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.792628 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:42.792649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:42.794758 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:42.797733 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:42.797745 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:42.797758 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:42.798147 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:42.798547 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:42.798553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:42.798557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:42.798739 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:42.799248 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:42.799255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:42.799584 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 11:47:42.799841 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 11:47:42.800342 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":255,"threads_started":1}
I20260924 11:47:42.802873 5918 heartbeater.cc:347] Connected to a master server at 127.1.210.62:38691
I20260924 11:47:42.803190 5918 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:42.803969 5918 heartbeater.cc:511] Master 127.1.210.62:38691 requested a full tablet report, sending...
I20260924 11:47:42.805305 5462 ts_manager.cc:194] Registered new tserver with Master: 65765790a8db4ac0b25f3d4f02600183 (127.1.210.3:36759)
I20260924 11:47:42.806052 5462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:33423
I20260924 11:47:42.815490 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260924 11:47:42.846545 5651 heartbeater.cc:503] Master 127.1.210.62:38691 was elected leader, sending a full tablet report...
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:42.862545 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.848192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.1.210.1:35635)
0924 11:47:42.848800 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:42.848808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:42.848837 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:47:42.848904 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:42.848910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:42.849178 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:42.849306 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:42.850531 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.851506 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.852629 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.853198 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.853297 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.856731 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:42.856741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:42.856756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:42.856789 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:42.859678 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.860370 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.860382 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.860385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.860559 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.860940 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.860946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.860949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.861358 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:42.861659 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:42.861960 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 11:47:42.862207 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":334,"threads_started":1}
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:42.878890 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.866256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.1.210.62:38691)
0924 11:47:42.866456 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:42.866464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:42.866669 (+ 205us) server_negotiation.cc:407] Connection header received
0924 11:47:42.866848 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:42.866853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:42.866921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:42.867010 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:42.868111 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.868815 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.870323 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.870730 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.870863 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.874542 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:42.874546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:42.874550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:42.874587 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:42.876786 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.877396 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.877400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.877403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.877526 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.877825 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.877828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.877830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.878106 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:42.878306 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:42.878532 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 11:47:42.878676 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 11:47:42.879418 5462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51546
I20260924 11:47:42.880311 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5523
I20260924 11:47:42.891528 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.881266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.1.210.1:35635)
0924 11:47:42.881468 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:42.881476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:42.881596 (+ 120us) server_negotiation.cc:407] Connection header received
0924 11:47:42.881709 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:42.881712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:42.881795 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:42.881871 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:42.882916 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.883558 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.884680 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.885056 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.885193 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:42.887238 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:42.887242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:42.887245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:42.887272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:42.889625 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.890175 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.890178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.890181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.890282 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:42.890567 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:42.890570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:42.890572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:42.890790 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:42.890954 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:42.891167 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 11:47:42.891326 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:47:42.892079 5646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:42.975910 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5523
I20260924 11:47:42.991408 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5657
Sep 24 11:47:42 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:43.007277 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:42.992286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.1.210.2:37547)
0924 11:47:42.992759 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:42.992766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:42.992790 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:47:42.992853 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:42.992859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:42.993120 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:42.993292 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:42.994433 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.995396 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.996696 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:42.997424 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:42.997568 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:43.001951 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:43.001960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:43.001973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:43.002004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:43.004804 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:43.005555 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:43.005563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:43.005565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:43.005685 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:43.006013 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:43.006016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:43.006018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:43.006317 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:43.006524 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:43.006822 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 11:47:43.007009 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":236,"threads_started":1}
I20260924 11:47:43.008474 5780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:43.087975 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5657
I20260924 11:47:43.104370 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5790
Sep 24 11:47:43 dist-test-slave-dd32 krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:43.120814 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:43.105410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.1.210.3:36759)
0924 11:47:43.106037 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:43.106044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:43.106072 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:47:43.106130 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:43.106136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:43.106357 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:43.106479 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:43.107625 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:43.108676 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:43.109942 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:43.110506 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:43.110608 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:43.115029 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:43.115040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:43.115054 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:43.115083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:43.117950 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:43.118611 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:43.118618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:43.118620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:43.118741 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:43.119187 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:43.119190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:43.119192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:43.119563 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:43.119877 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:43.120235 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 11:47:43.120482 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":334,"threads_started":1}
I20260924 11:47:43.122099 5913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:43.202657 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5790
I20260924 11:47:43.217905 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5430
I20260924 11:47:43.219004 5492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:43.318819 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5430
2026-09-24T11:47:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3749 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:43 dist-test-slave-dd32 krb5kdc[5937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:43 dist-test-slave-dd32 krb5kdc[5937](info): set up 2 sockets
Sep 24 11:47:43 dist-test-slave-dd32 krb5kdc[5937](info): commencing operation
krb5kdc: starting...
W20260924 11:47:44.401439 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:44 dist-test-slave-dd32 krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:44.588742 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36683
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:44.902751 5950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:44.903033 5950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:44.903123 5950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:44.910193 5950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:44.910301 5950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:44.910346 5950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:44.910403 5950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:44.910449 5950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:44.920035 5950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:36683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:44.921249 5950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:44.922504 5950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:44.936029 5958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:44.936021 5955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:44.936069 5956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:44.942649 5950 server_base.cc:1034] running on GCE node
I20260924 11:47:44.943404 5950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:44.944815 5950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:44.946012 5950 hybrid_clock.cc:648] HybridClock initialized: now 1790250464945972 us; error 30 us; skew 500 ppm
I20260924 11:47:44.946376 5950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:44 dist-test-slave-dd32 krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:44.949378 5950 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:44.950552 5950 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:44.950639 5950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:44.950891 5950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:44.953750 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "88450e70150f4a7ca9dd5643df51bda7"
format_stamp: "Formatted at 2026-09-24 11:47:44 on dist-test-slave-dd32"
server_key: "9c364b8c62a0b65d239bdd932390861b"
server_key_iv: "41216822aa521e2bd8dc4d2baee2ce35"
server_key_version: "encryptionkey@0"
I20260924 11:47:44.954461 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "88450e70150f4a7ca9dd5643df51bda7"
format_stamp: "Formatted at 2026-09-24 11:47:44 on dist-test-slave-dd32"
server_key: "9c364b8c62a0b65d239bdd932390861b"
server_key_iv: "41216822aa521e2bd8dc4d2baee2ce35"
server_key_version: "encryptionkey@0"
I20260924 11:47:44.959640 5950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:47:44.963177 5964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:44.964469 5950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:44.964680 5950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "88450e70150f4a7ca9dd5643df51bda7"
format_stamp: "Formatted at 2026-09-24 11:47:44 on dist-test-slave-dd32"
server_key: "9c364b8c62a0b65d239bdd932390861b"
server_key_iv: "41216822aa521e2bd8dc4d2baee2ce35"
server_key_version: "encryptionkey@0"
I20260924 11:47:44.964915 5950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:44.993098 5950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:44.997593 5950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:44.997898 5950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:45.016945 5950 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:36683
I20260924 11:47:45.016920 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:36683 every 8 connection(s)
I20260924 11:47:45.017947 5950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:45.021557 6016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:45.026652 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 5950
I20260924 11:47:45.026893 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:45.027526 1864 external_mini_cluster.cc:1428] Setting key b61c61a6488a9c7709b1f7b909baac31
I20260924 11:47:45.032473 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7: Bootstrap starting.
I20260924 11:47:45.035385 6016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:45.036329 6016 log.cc:824] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:45.039000 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7: No bootstrap required, opened a new log
Sep 24 11:47:45 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:45.046841 6016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88450e70150f4a7ca9dd5643df51bda7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36683 } }
I20260924 11:47:45.047194 6016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:45.047324 6016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88450e70150f4a7ca9dd5643df51bda7, State: Initialized, Role: FOLLOWER
I20260924 11:47:45.047927 6016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88450e70150f4a7ca9dd5643df51bda7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36683 } }
I20260924 11:47:45.048183 6016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:45.048326 6016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:45.048509 6016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:45.048887 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:45.029740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.1.210.62:36683)
0924 11:47:45.030488 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:45.030503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:45.030542 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:47:45.032154 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:45.032174 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:45.032537 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:45.032949 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:45.034452 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:45.035798 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:45.037124 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:45.037768 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:45.037951 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:45.041973 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:45.041980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:45.041996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:45.042027 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:45.044972 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:45.045612 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:45.045618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:45.045624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:45.045778 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:45.046155 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:45.046159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:45.046161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:45.046629 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:45.046914 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:45.047355 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 11:47:45.047773 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":330,"threads_started":1}
I20260924 11:47:45.051301 6016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88450e70150f4a7ca9dd5643df51bda7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36683 } }
I20260924 11:47:45.051847 6016 leader_election.cc:302] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88450e70150f4a7ca9dd5643df51bda7; no voters:
I20260924 11:47:45.052359 6016 leader_election.cc:288] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:45.052490 6021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:45.053174 6021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [term 1 LEADER]: Becoming Leader. State: Replica: 88450e70150f4a7ca9dd5643df51bda7, State: Running, Role: LEADER
I20260924 11:47:45.053629 6021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88450e70150f4a7ca9dd5643df51bda7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36683 } }
I20260924 11:47:45.054288 6016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:45.056239 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88450e70150f4a7ca9dd5643df51bda7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88450e70150f4a7ca9dd5643df51bda7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36683 } } }
I20260924 11:47:45.056324 6023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88450e70150f4a7ca9dd5643df51bda7. Latest consensus state: current_term: 1 leader_uuid: "88450e70150f4a7ca9dd5643df51bda7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88450e70150f4a7ca9dd5643df51bda7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36683 } } }
I20260924 11:47:45.056708 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:45.056756 6023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:45.057965 6029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:45.062949 6029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:45.071182 6029 catalog_manager.cc:1347] Generated new cluster ID: 228abeaab82d41df89b2e4a89956f7c6
I20260924 11:47:45.071288 6029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:45.095991 6029 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:45.097839 6029 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:45.110944 6029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7: Generated new TSK 0
I20260924 11:47:45.111979 6029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:45.160353 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36683
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:45.473675 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:45.473939 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:45.474128 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:45.481170 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:45.481263 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:45.481475 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:45.490746 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:45.491968 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:45.493235 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:45.505661 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:45.505721 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:45.505745 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:45.506527 6042 server_base.cc:1034] running on GCE node
I20260924 11:47:45.507129 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:45.508090 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:45.509300 6042 hybrid_clock.cc:648] HybridClock initialized: now 1790250465509251 us; error 43 us; skew 500 ppm
I20260924 11:47:45.509565 6042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:45 dist-test-slave-dd32 krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:45.512444 6042 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:45.513577 6042 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:45.513710 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:45.514011 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:45.516757 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "74b6b58c2f654ed68268220addae6ba8"
format_stamp: "Formatted at 2026-09-24 11:47:45 on dist-test-slave-dd32"
server_key: "5adbc3c6d9b51dcd78fc117edcd9522c"
server_key_iv: "abd978a888b62084ab952eceb7747e2c"
server_key_version: "encryptionkey@0"
I20260924 11:47:45.517690 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "74b6b58c2f654ed68268220addae6ba8"
format_stamp: "Formatted at 2026-09-24 11:47:45 on dist-test-slave-dd32"
server_key: "5adbc3c6d9b51dcd78fc117edcd9522c"
server_key_iv: "abd978a888b62084ab952eceb7747e2c"
server_key_version: "encryptionkey@0"
I20260924 11:47:45.522292 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:47:45.525557 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:45.526809 6042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:47:45.526975 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "74b6b58c2f654ed68268220addae6ba8"
format_stamp: "Formatted at 2026-09-24 11:47:45 on dist-test-slave-dd32"
server_key: "5adbc3c6d9b51dcd78fc117edcd9522c"
server_key_iv: "abd978a888b62084ab952eceb7747e2c"
server_key_version: "encryptionkey@0"
I20260924 11:47:45.527175 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:45.544759 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:45.548609 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:45.548857 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:45.549834 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:45.551080 6042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:45.551188 6042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:45.551326 6042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:45.551419 6042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:45.583576 6042 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:36101
I20260924 11:47:45.583626 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:36101 every 8 connection(s)
I20260924 11:47:45.584401 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:45.589344 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6042
I20260924 11:47:45.589551 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:45.590222 1864 external_mini_cluster.cc:1428] Setting key 70f1e9ecf39f37e752d63b54f6f37806
Sep 24 11:47:45 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:45.604009 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:45.586831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:53561 (local address 127.1.210.62:36683)
0924 11:47:45.587260 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:45.587266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:45.588216 (+ 950us) server_negotiation.cc:407] Connection header received
0924 11:47:45.590071 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:45.590079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:45.590167 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:45.590297 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:45.591909 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:45.592950 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:45.594489 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:45.595185 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:45.595324 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:45.598554 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:45.598559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:45.598562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:45.598591 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:45.600985 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:45.601894 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:45.601898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:45.601900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:45.602012 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:45.602464 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:45.602467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:45.602469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:45.602691 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:45.602811 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:45.603611 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 11:47:45.603755 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":211,"threads_started":1}
I20260924 11:47:45.605535 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:45.587395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36683 (local address 127.1.210.1:53561)
0924 11:47:45.588058 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:45.588100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:45.589810 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:45.590527 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:45.590535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:45.591178 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:45.591740 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:45.591765 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:45.593152 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:45.593156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:45.594352 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:45.594362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:45.595371 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:45.595375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:45.595555 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:45.596183 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:45.596202 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:45.598368 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:45.601135 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:45.601145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:45.601163 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:45.601782 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:45.602120 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:45.602127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:45.602134 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:45.602361 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:45.602822 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:45.602832 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:45.603239 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 11:47:45.603536 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 11:47:45.604073 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
I20260924 11:47:45.607105 6169 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36683
I20260924 11:47:45.607395 6169 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:45.608012 6169 heartbeater.cc:511] Master 127.1.210.62:36683 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:47:45.610085 5981 ts_manager.cc:194] Registered new tserver with Master: 74b6b58c2f654ed68268220addae6ba8 (127.1.210.1:36101)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:45.611722 5981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:53561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:45.634239 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36683
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:45.944394 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:45.944695 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:45.944878 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:45.951968 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:45.952112 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:45.952299 6175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:45.961861 6175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:45.963088 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:45.964372 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:45.976934 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:45.976938 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:45.976907 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:45.977661 6175 server_base.cc:1034] running on GCE node
I20260924 11:47:45.978207 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:45.979100 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:45.980324 6175 hybrid_clock.cc:648] HybridClock initialized: now 1790250465980285 us; error 48 us; skew 500 ppm
I20260924 11:47:45.980616 6175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:45 dist-test-slave-dd32 krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:45.983521 6175 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:45.984503 6175 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:45.984586 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:45.984830 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:45.987550 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "b9e0926e68004344a00a76552e6483a5"
format_stamp: "Formatted at 2026-09-24 11:47:45 on dist-test-slave-dd32"
server_key: "1e07363da7fa98021227fefcb22057f5"
server_key_iv: "ce6bcd2d22e7269565ec757f3e9b3bd9"
server_key_version: "encryptionkey@0"
I20260924 11:47:45.988261 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "b9e0926e68004344a00a76552e6483a5"
format_stamp: "Formatted at 2026-09-24 11:47:45 on dist-test-slave-dd32"
server_key: "1e07363da7fa98021227fefcb22057f5"
server_key_iv: "ce6bcd2d22e7269565ec757f3e9b3bd9"
server_key_version: "encryptionkey@0"
I20260924 11:47:45.992795 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 11:47:45.995885 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:45.997174 6175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:47:45.997339 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "b9e0926e68004344a00a76552e6483a5"
format_stamp: "Formatted at 2026-09-24 11:47:45 on dist-test-slave-dd32"
server_key: "1e07363da7fa98021227fefcb22057f5"
server_key_iv: "ce6bcd2d22e7269565ec757f3e9b3bd9"
server_key_version: "encryptionkey@0"
I20260924 11:47:45.997534 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:46.012864 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:46.016585 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:46.016850 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:46.017777 6175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:46.019011 6175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:46.019093 6175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:46.019243 6175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:46.019342 6175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:46.053858 6175 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39009
I20260924 11:47:46.053951 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39009 every 8 connection(s)
I20260924 11:47:46.054713 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:46.061237 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6175
I20260924 11:47:46.061441 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:46.061883 1864 external_mini_cluster.cc:1428] Setting key 342d1c178dd0b228380dd4d6980a7ddf
Sep 24 11:47:46 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:46.074103 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.057156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57097 (local address 127.1.210.62:36683)
0924 11:47:46.057350 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.057356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.058596 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 11:47:46.060343 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.060347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.060425 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.060511 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:46.062101 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.062835 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.064340 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.064734 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.064866 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.068356 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:46.068362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:46.068366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:46.068396 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:46.071030 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.071956 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.071960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.071962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.072073 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.072518 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.072521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.072523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.072742 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:46.072852 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.073709 (+ 857us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.073874 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:47:46.075598 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.057735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36683 (local address 127.1.210.2:57097)
0924 11:47:46.058447 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.058487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.060109 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.060699 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.060706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.061260 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:46.061902 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.061943 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.062982 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.062988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.064186 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.064197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.064952 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.064960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.065241 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.065954 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:46.065995 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:46.068105 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:46.071164 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:46.071175 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:46.071194 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:46.071841 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:46.072175 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:46.072182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:46.072188 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:46.072411 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:46.072873 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:46.072884 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:46.073341 (+ 457us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.073637 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.074181 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":323,"threads_started":1}
I20260924 11:47:46.076958 6302 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36683
I20260924 11:47:46.077278 6302 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:46.077903 6302 heartbeater.cc:511] Master 127.1.210.62:36683 requested a full tablet report, sending...
I20260924 11:47:46.079190 5981 ts_manager.cc:194] Registered new tserver with Master: b9e0926e68004344a00a76552e6483a5 (127.1.210.2:39009)
I20260924 11:47:46.079989 5981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:57097
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:46.104241 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36683
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:46.417227 6307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:46.417481 6307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:46.417584 6307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:46.424438 6307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:46.424537 6307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:46.424649 6307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:46.434186 6307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:46.435340 6307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:46.436630 6307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:46.448973 6312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:46.449033 6315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:46.449035 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:46.449843 6307 server_base.cc:1034] running on GCE node
I20260924 11:47:46.450461 6307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:46.451339 6307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:46.452557 6307 hybrid_clock.cc:648] HybridClock initialized: now 1790250466452519 us; error 52 us; skew 500 ppm
I20260924 11:47:46.452864 6307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:46 dist-test-slave-dd32 krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:46.459908 6307 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:46.461114 6307 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:46.461251 6307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:46.461581 6307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:46.464329 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "f9d0243417724dc48c03ff5823e41be3"
format_stamp: "Formatted at 2026-09-24 11:47:46 on dist-test-slave-dd32"
server_key: "45479ccb353cfa30d41e9bb6a49b177d"
server_key_iv: "b43d7789a041e2c5975977064eb7a0bc"
server_key_version: "encryptionkey@0"
I20260924 11:47:46.465234 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "f9d0243417724dc48c03ff5823e41be3"
format_stamp: "Formatted at 2026-09-24 11:47:46 on dist-test-slave-dd32"
server_key: "45479ccb353cfa30d41e9bb6a49b177d"
server_key_iv: "b43d7789a041e2c5975977064eb7a0bc"
server_key_version: "encryptionkey@0"
I20260924 11:47:46.470167 6307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:47:46.473589 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:46.474926 6307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:47:46.475096 6307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "f9d0243417724dc48c03ff5823e41be3"
format_stamp: "Formatted at 2026-09-24 11:47:46 on dist-test-slave-dd32"
server_key: "45479ccb353cfa30d41e9bb6a49b177d"
server_key_iv: "b43d7789a041e2c5975977064eb7a0bc"
server_key_version: "encryptionkey@0"
I20260924 11:47:46.475318 6307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:46.502374 6307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:46.506728 6307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:46.507006 6307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:46.507963 6307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:46.509315 6307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:46.509439 6307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:46.509552 6307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:46.509635 6307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:46.540864 6307 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:41129
I20260924 11:47:46.540948 6433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:41129 every 8 connection(s)
I20260924 11:47:46.541765 6307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:46.542205 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6307
I20260924 11:47:46.542426 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:46.542990 1864 external_mini_cluster.cc:1428] Setting key 6f6db6e11f16d01afe34b19c8eb13d57
Sep 24 11:47:46 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:46.560685 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.544212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:33937 (local address 127.1.210.62:36683)
0924 11:47:46.544413 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.544419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.545758 (+ 1339us) server_negotiation.cc:407] Connection header received
0924 11:47:46.547656 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.547662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.547745 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.547856 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:46.549325 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.550152 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.551583 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.551961 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.552049 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.555196 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:46.555201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:46.555204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:46.555233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:46.558252 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.558909 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.558912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.558914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.559017 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.559408 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.559411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.559413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.559651 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:46.559755 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.560320 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.560461 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:47:46.561631 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.544823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36683 (local address 127.1.210.3:33937)
0924 11:47:46.545603 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.545645 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.547390 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.548081 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.548089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.548563 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:46.549127 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.549143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.550278 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.550282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.551467 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.551478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.552126 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.552131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.552307 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.552871 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:46.552891 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:46.555003 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:46.558365 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:46.558371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:46.558381 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:46.558811 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:46.559137 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:46.559140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:46.559142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:46.559315 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:46.559750 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:46.559755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:46.560021 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.560235 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.560572 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":357,"threads_started":1}
I20260924 11:47:46.562887 6434 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36683
I20260924 11:47:46.563156 6434 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:46.563751 6434 heartbeater.cc:511] Master 127.1.210.62:36683 requested a full tablet report, sending...
I20260924 11:47:46.565040 5981 ts_manager.cc:194] Registered new tserver with Master: f9d0243417724dc48c03ff5823e41be3 (127.1.210.3:41129)
I20260924 11:47:46.565860 5981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:33937
I20260924 11:47:46.570283 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:47:46.586479 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.574596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.1.210.62:36683)
0924 11:47:46.574739 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.574743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.575292 (+ 549us) server_negotiation.cc:407] Connection header received
0924 11:47:46.575416 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.575419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.575468 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.575541 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:46.576847 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.577567 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.578875 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.579312 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.579441 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.581592 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:46.581596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:46.581599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:46.581626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:46.583863 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.584447 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.584454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.584457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.584578 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.584935 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.584942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.584947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.585312 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:46.585543 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.585866 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.586067 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:47:46.589932 5981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:46.596303 5981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:46.615406 6169 heartbeater.cc:503] Master 127.1.210.62:36683 was elected leader, sending a full tablet report...
I20260924 11:47:46.629923 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.1.210.3:41129)
0924 11:47:46.621106 (+ 13665us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.621117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.621157 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:47:46.621257 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.621265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.621543 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.621772 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:46.623075 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.624671 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.627532 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.628701 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.628846 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.629102 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.629411 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.629582 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13425,"thread_start_us":14598,"threads_started":1}
I20260924 11:47:46.630582 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.607209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:41129 (local address 127.0.0.1:44284)
0924 11:47:46.620106 (+ 12897us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.620155 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.620377 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.622329 (+ 1952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.622338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.622374 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:46.622858 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.622875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.624822 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.624828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.627359 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.627377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.628913 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.628920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.629152 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.629188 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.629344 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.629519 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12567,"mutex_wait_us":1,"thread_start_us":15367,"threads_started":1}
I20260924 11:47:46.630846 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.607362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36101 (local address 127.0.0.1:43692)
0924 11:47:46.622894 (+ 15532us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.622915 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.623027 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.623590 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.623595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.623619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:46.624007 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.624019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.625964 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.625970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.628095 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.628108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.630010 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.630020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.630181 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.630215 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.630321 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.630445 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15371,"thread_start_us":12261,"threads_started":1}
I20260924 11:47:46.630908 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.608191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.1.210.1:36101)
0924 11:47:46.622712 (+ 14521us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.622723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.623049 (+ 326us) server_negotiation.cc:407] Connection header received
0924 11:47:46.623155 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.623165 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.623388 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.624208 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:46.624421 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.625850 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.628257 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.629701 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.630006 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.630226 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.630368 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.630512 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14361,"thread_start_us":14513,"threads_started":1}
I20260924 11:47:46.631883 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.623772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39009 (local address 127.0.0.1:39722)
0924 11:47:46.624306 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.624319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.624447 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.624958 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.624963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.625002 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:46.625426 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.625437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.627212 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.627217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.629021 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.629035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.631143 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.631148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.631351 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.631388 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.631474 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.631587 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":307,"threads_started":1}
I20260924 11:47:46.631978 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.623218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.1.210.2:39009)
0924 11:47:46.623702 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.623710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.624409 (+ 699us) server_negotiation.cc:407] Connection header received
0924 11:47:46.624602 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.624607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.624771 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.624957 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:46.625603 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.627050 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.629256 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.630698 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.630826 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.631027 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.631518 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.631658 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":278,"threads_started":1}
I20260924 11:47:46.634563 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 2d02184f0ce04cd1b725d72af5572221 (DEFAULT_TABLE table=test-table [id=2b6f32b8dc9b4d5092ea0a43831e859f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:46.634617 6369 tablet_service.cc:1467] Processing CreateTablet for tablet 2d02184f0ce04cd1b725d72af5572221 (DEFAULT_TABLE table=test-table [id=2b6f32b8dc9b4d5092ea0a43831e859f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:46.635762 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d02184f0ce04cd1b725d72af5572221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:46.635957 6369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d02184f0ce04cd1b725d72af5572221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:46.636186 6237 tablet_service.cc:1467] Processing CreateTablet for tablet 2d02184f0ce04cd1b725d72af5572221 (DEFAULT_TABLE table=test-table [id=2b6f32b8dc9b4d5092ea0a43831e859f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:46.637436 6237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d02184f0ce04cd1b725d72af5572221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:46.646471 6452 tablet_bootstrap.cc:492] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Bootstrap starting.
I20260924 11:47:46.649616 6453 tablet_bootstrap.cc:492] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Bootstrap starting.
I20260924 11:47:46.649997 6452 tablet_bootstrap.cc:654] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:46.651325 6452 log.cc:824] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:46.651432 6454 tablet_bootstrap.cc:492] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Bootstrap starting.
I20260924 11:47:46.653246 6453 tablet_bootstrap.cc:654] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:46.654300 6454 tablet_bootstrap.cc:654] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:46.654429 6453 log.cc:824] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:46.655436 6454 log.cc:824] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:46.655756 6452 tablet_bootstrap.cc:492] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: No bootstrap required, opened a new log
I20260924 11:47:46.656013 6452 ts_tablet_manager.cc:1397] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 11:47:46.657372 6453 tablet_bootstrap.cc:492] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: No bootstrap required, opened a new log
I20260924 11:47:46.657617 6453 ts_tablet_manager.cc:1397] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260924 11:47:46.657966 6454 tablet_bootstrap.cc:492] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: No bootstrap required, opened a new log
I20260924 11:47:46.658250 6454 ts_tablet_manager.cc:1397] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 11:47:46.664877 6454 raft_consensus.cc:348] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.665285 6454 raft_consensus.cc:374] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:46.665393 6454 raft_consensus.cc:729] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e0926e68004344a00a76552e6483a5, State: Initialized, Role: FOLLOWER
I20260924 11:47:46.665977 6454 consensus_queue.cc:260] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.666464 6452 raft_consensus.cc:348] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.666893 6452 raft_consensus.cc:374] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:46.666918 6302 heartbeater.cc:503] Master 127.1.210.62:36683 was elected leader, sending a full tablet report...
I20260924 11:47:46.666990 6452 raft_consensus.cc:729] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b6b58c2f654ed68268220addae6ba8, State: Initialized, Role: FOLLOWER
I20260924 11:47:46.667543 6452 consensus_queue.cc:260] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.667762 6453 raft_consensus.cc:348] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.668273 6453 raft_consensus.cc:374] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:46.669209 6452 ts_tablet_manager.cc:1428] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 11:47:46.668408 6453 raft_consensus.cc:729] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d0243417724dc48c03ff5823e41be3, State: Initialized, Role: FOLLOWER
I20260924 11:47:46.669919 6454 ts_tablet_manager.cc:1428] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260924 11:47:46.670107 6453 consensus_queue.cc:260] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.671658 6453 ts_tablet_manager.cc:1428] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260924 11:47:46.672076 6434 heartbeater.cc:503] Master 127.1.210.62:36683 was elected leader, sending a full tablet report...
I20260924 11:47:46.686861 6459 raft_consensus.cc:482] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:46.687134 6459 raft_consensus.cc:504] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.688695 6459 leader_election.cc:288] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d0243417724dc48c03ff5823e41be3 (127.1.210.3:41129), b9e0926e68004344a00a76552e6483a5 (127.1.210.2:39009)
I20260924 11:47:46.695171 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.688996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:40791 (local address 127.1.210.3:41129)
0924 11:47:46.689204 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.689208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.689655 (+ 447us) server_negotiation.cc:407] Connection header received
0924 11:47:46.689987 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.689992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.690061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.690122 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:46.690769 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.691486 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.693454 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.694176 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.694330 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.694460 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.694793 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.694910 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:47:46.695282 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.689738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:48695 (local address 127.1.210.2:39009)
0924 11:47:46.689983 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.689988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.690286 (+ 298us) server_negotiation.cc:407] Connection header received
0924 11:47:46.690484 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.690489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.690553 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.690646 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:47:46.691116 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.691811 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.693591 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.694326 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.694508 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.694603 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.694922 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.695037 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 11:47:46.695444 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.688923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:41129 (local address 127.1.210.1:40791)
0924 11:47:46.689563 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.689586 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.689768 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.690296 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.690299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.690319 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:46.690634 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.690641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.691610 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.691618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.693306 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.693317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.694390 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.694396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.694570 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.694636 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.694770 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.694908 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":93,"thread_start_us":339,"threads_started":1}
I20260924 11:47:46.695533 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.689645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39009 (local address 127.1.210.1:48695)
0924 11:47:46.690185 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:46.690202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:46.690354 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:46.690660 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:46.690664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:46.690687 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:47:46.690981 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.690988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.691942 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.691948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.693449 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:46.693460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.694573 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.694579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:46.694776 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.694799 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:47:46.694899 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:47:46.695016 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":3,"thread_start_us":283,"threads_started":1}
I20260924 11:47:46.696377 6257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d02184f0ce04cd1b725d72af5572221" candidate_uuid: "74b6b58c2f654ed68268220addae6ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e0926e68004344a00a76552e6483a5" is_pre_election: true
I20260924 11:47:46.696378 6389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d02184f0ce04cd1b725d72af5572221" candidate_uuid: "74b6b58c2f654ed68268220addae6ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d0243417724dc48c03ff5823e41be3" is_pre_election: true
I20260924 11:47:46.696830 6257 raft_consensus.cc:2454] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b6b58c2f654ed68268220addae6ba8 in term 0.
I20260924 11:47:46.696839 6389 raft_consensus.cc:2454] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b6b58c2f654ed68268220addae6ba8 in term 0.
I20260924 11:47:46.697541 6058 leader_election.cc:302] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b6b58c2f654ed68268220addae6ba8, f9d0243417724dc48c03ff5823e41be3; no voters:
I20260924 11:47:46.697893 6459 raft_consensus.cc:2781] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:46.698053 6459 raft_consensus.cc:482] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:46.698176 6459 raft_consensus.cc:3037] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:46.700280 6459 raft_consensus.cc:504] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.700851 6459 leader_election.cc:288] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [CANDIDATE]: Term 1 election: Requested vote from peers f9d0243417724dc48c03ff5823e41be3 (127.1.210.3:41129), b9e0926e68004344a00a76552e6483a5 (127.1.210.2:39009)
I20260924 11:47:46.701297 6389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d02184f0ce04cd1b725d72af5572221" candidate_uuid: "74b6b58c2f654ed68268220addae6ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d0243417724dc48c03ff5823e41be3"
I20260924 11:47:46.701489 6389 raft_consensus.cc:3037] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:46.701467 6257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d02184f0ce04cd1b725d72af5572221" candidate_uuid: "74b6b58c2f654ed68268220addae6ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e0926e68004344a00a76552e6483a5"
I20260924 11:47:46.701673 6257 raft_consensus.cc:3037] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:46.703656 6389 raft_consensus.cc:2454] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b6b58c2f654ed68268220addae6ba8 in term 1.
I20260924 11:47:46.703879 6257 raft_consensus.cc:2454] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b6b58c2f654ed68268220addae6ba8 in term 1.
I20260924 11:47:46.704032 6058 leader_election.cc:302] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b6b58c2f654ed68268220addae6ba8, f9d0243417724dc48c03ff5823e41be3; no voters:
I20260924 11:47:46.704324 6459 raft_consensus.cc:2781] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:46.704743 6459 raft_consensus.cc:686] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 74b6b58c2f654ed68268220addae6ba8, State: Running, Role: LEADER
I20260924 11:47:46.705292 6459 consensus_queue.cc:237] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } }
I20260924 11:47:46.709658 5981 catalog_manager.cc:5495] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74b6b58c2f654ed68268220addae6ba8 (127.1.210.1). New cstate: current_term: 1 leader_uuid: "74b6b58c2f654ed68268220addae6ba8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b6b58c2f654ed68268220addae6ba8" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:46.742113 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.736888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.1.210.1:36101)
0924 11:47:46.737099 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.737107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.737159 (+ 52us) server_negotiation.cc:407] Connection header received
0924 11:47:46.737429 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.737433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.737504 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.737654 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:47:46.738009 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.738694 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.740028 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.740434 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.740563 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.741008 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:47:46.741552 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:47:46.741707 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.741785 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.741858 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:47:46.749723 6257 raft_consensus.cc:1261] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 1 FOLLOWER]: Refusing update from remote peer 74b6b58c2f654ed68268220addae6ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:46.749723 6389 raft_consensus.cc:1261] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 1 FOLLOWER]: Refusing update from remote peer 74b6b58c2f654ed68268220addae6ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:46.750660 6463 consensus_queue.cc:1035] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d0243417724dc48c03ff5823e41be3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 41129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:46.751034 6459 consensus_queue.cc:1035] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e0926e68004344a00a76552e6483a5" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:46.763938 6468 mvcc.cc:205] Tried to move back new op lower bound from 7332865911797243904 to 7332865911629107200. Current Snapshot: MvccSnapshot[applied={T|T < 7332865911797243904}]
I20260924 11:47:46.775804 5980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:46.776084 5980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:46.779565 5980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88450e70150f4a7ca9dd5643df51bda7: Sending DeleteTablet for 3 replicas of tablet 2d02184f0ce04cd1b725d72af5572221
I20260924 11:47:46.780730 6104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d02184f0ce04cd1b725d72af5572221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:46 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:43692
I20260924 11:47:46.780722 6237 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d02184f0ce04cd1b725d72af5572221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:46 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:39722
I20260924 11:47:46.780864 6369 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d02184f0ce04cd1b725d72af5572221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:46 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:44284
I20260924 11:47:46.782761 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6042
I20260924 11:47:46.794476 6478 tablet_replica.cc:330] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: stopping tablet replica
I20260924 11:47:46.794945 6478 raft_consensus.cc:2229] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:46.795512 6478 raft_consensus.cc:2258] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:46.797577 6478 ts_tablet_manager.cc:1905] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:46.804776 6478 ts_tablet_manager.cc:1918] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:46.805012 6478 log.cc:1196] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/2d02184f0ce04cd1b725d72af5572221
I20260924 11:47:46.805657 6478 ts_tablet_manager.cc:1939] T 2d02184f0ce04cd1b725d72af5572221 P f9d0243417724dc48c03ff5823e41be3: Deleting consensus metadata
I20260924 11:47:46.807528 5966 catalog_manager.cc:4841] TS f9d0243417724dc48c03ff5823e41be3 (127.1.210.3:41129): tablet 2d02184f0ce04cd1b725d72af5572221 (table test-table [id=2b6f32b8dc9b4d5092ea0a43831e859f]) successfully deleted
I20260924 11:47:46.809787 6477 tablet_replica.cc:330] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: stopping tablet replica
I20260924 11:47:46.810452 6477 raft_consensus.cc:2229] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:46.811327 6477 raft_consensus.cc:2258] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:46.812337 6476 tablet_replica.cc:330] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: stopping tablet replica
I20260924 11:47:46.812765 6476 raft_consensus.cc:2229] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:46.813334 6476 raft_consensus.cc:2258] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:46.814146 6477 ts_tablet_manager.cc:1905] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:46.815115 6476 ts_tablet_manager.cc:1905] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:47:46 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:46.821897 6476 ts_tablet_manager.cc:1918] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:46.822139 6476 log.cc:1196] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/2d02184f0ce04cd1b725d72af5572221
I20260924 11:47:46.822357 6477 ts_tablet_manager.cc:1918] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:46.822757 6476 ts_tablet_manager.cc:1939] T 2d02184f0ce04cd1b725d72af5572221 P b9e0926e68004344a00a76552e6483a5: Deleting consensus metadata
I20260924 11:47:46.822569 6477 log.cc:1196] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/2d02184f0ce04cd1b725d72af5572221
I20260924 11:47:46.823473 6477 ts_tablet_manager.cc:1939] T 2d02184f0ce04cd1b725d72af5572221 P 74b6b58c2f654ed68268220addae6ba8: Deleting consensus metadata
I20260924 11:47:46.824546 5966 catalog_manager.cc:4841] TS b9e0926e68004344a00a76552e6483a5 (127.1.210.2:39009): tablet 2d02184f0ce04cd1b725d72af5572221 (table test-table [id=2b6f32b8dc9b4d5092ea0a43831e859f]) successfully deleted
I20260924 11:47:46.825147 5967 catalog_manager.cc:4841] TS 74b6b58c2f654ed68268220addae6ba8 (127.1.210.1:36101): tablet 2d02184f0ce04cd1b725d72af5572221 (table test-table [id=2b6f32b8dc9b4d5092ea0a43831e859f]) successfully deleted
I20260924 11:47:46.828052 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.784425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.1.210.1:36101)
0924 11:47:46.784655 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.784664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.808743 (+ 24079us) server_negotiation.cc:407] Connection header received
0924 11:47:46.808972 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.808979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.809101 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.809225 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:46.810926 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.812144 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.814495 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.815210 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.815318 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.820521 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:46.820535 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:46.820553 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:46.820598 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:46.825234 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.826096 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.826103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.826105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.826238 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.826629 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.826632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.826634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.826963 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:46.827246 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.827555 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.827728 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:47:46.828801 6164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:46.925534 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6042
I20260924 11:47:46.946452 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6175
Sep 24 11:47:46 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:46.960162 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:46.947289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.1.210.2:39009)
0924 11:47:46.947454 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:46.947458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:46.947475 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:47:46.947558 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:46.947562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:46.947626 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:46.947705 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:46.948891 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.949674 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.950863 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:46.951212 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:46.951299 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:46.955012 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:46.955023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:46.955037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:46.955068 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:46.957873 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.958443 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.958449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.958451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.958570 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:46.958884 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:46.958887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:46.958889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:46.959250 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:46.959520 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:46.959669 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 11:47:46.959822 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:47:46.960831 6297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:47.046584 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6175
I20260924 11:47:47.062668 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6307
Sep 24 11:47:47 dist-test-slave-dd32 krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:47.077040 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:47.063436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.1.210.3:41129)
0924 11:47:47.063609 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:47.063616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:47.063636 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:47:47.063843 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:47.063847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:47.063914 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:47.064017 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:47.065252 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:47.066033 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:47.067269 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:47.067657 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:47.067803 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:47.071857 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:47.071872 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:47.071888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:47.071917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:47.074385 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:47.075051 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:47.075058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:47.075060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:47.075189 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:47.075555 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:47.075562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:47.075565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:47.075915 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:47.076198 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:47.076524 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 11:47:47.076742 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:47:47.077723 6429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:47.167352 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6307
I20260924 11:47:47.183288 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 5950
I20260924 11:47:47.184453 6011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:47.288061 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 5950
2026-09-24T11:47:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3968 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T11:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:47Z Disabled control of system clock
I20260924 11:47:47.351487 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:33937
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:43141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 11:47:47.659452 6487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:47.659767 6487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:47.659881 6487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:47.667239 6487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:47.667349 6487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:47.667445 6487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:47.667553 6487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:47.667660 6487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:47.676791 6487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:43141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:33937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:47.678161 6487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:47.679991 6487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:47.691798 6492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:47.692097 6493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:47.692133 6495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:47.693356 6487 server_base.cc:1034] running on GCE node
I20260924 11:47:47.694186 6487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:47.695570 6487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:47.696798 6487 hybrid_clock.cc:648] HybridClock initialized: now 1790250467696755 us; error 40 us; skew 500 ppm
I20260924 11:47:47.697181 6487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:47.699322 6487 webserver.cc:460] Webserver started at http://127.1.210.62:42837/ using document root <none> and password file <none>
I20260924 11:47:47.699996 6487 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:47.700124 6487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:47.700439 6487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:47.703280 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "d1015fe22326442e84e5796339194665"
format_stamp: "Formatted at 2026-09-24 11:47:47 on dist-test-slave-dd32"
server_key: "d1b112153fefa7492014dded56e2d89d"
server_key_iv: "cee86370964d9cbbcffbf838c8d06a34"
server_key_version: "encryptionkey@0"
I20260924 11:47:47.704059 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "d1015fe22326442e84e5796339194665"
format_stamp: "Formatted at 2026-09-24 11:47:47 on dist-test-slave-dd32"
server_key: "d1b112153fefa7492014dded56e2d89d"
server_key_iv: "cee86370964d9cbbcffbf838c8d06a34"
server_key_version: "encryptionkey@0"
I20260924 11:47:47.708977 6487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 11:47:47.712227 6501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:47.713575 6487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:47:47.713752 6487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "d1015fe22326442e84e5796339194665"
format_stamp: "Formatted at 2026-09-24 11:47:47 on dist-test-slave-dd32"
server_key: "d1b112153fefa7492014dded56e2d89d"
server_key_iv: "cee86370964d9cbbcffbf838c8d06a34"
server_key_version: "encryptionkey@0"
I20260924 11:47:47.713958 6487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:47.719322 6487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:47.720052 6487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:47.720343 6487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:47.738250 6487 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:33937
I20260924 11:47:47.738283 6552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:33937 every 8 connection(s)
I20260924 11:47:47.739504 6487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:47.743069 6553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:47.747592 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6487
I20260924 11:47:47.747761 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:47.748380 1864 external_mini_cluster.cc:1428] Setting key fb9b383f15c58d630a3ef7c77cc8f2b7
I20260924 11:47:47.753710 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665: Bootstrap starting.
I20260924 11:47:47.756197 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:47.750271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.1.210.62:33937)
0924 11:47:47.751273 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:47.751284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:47.751326 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:47:47.752892 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:47.752912 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:47.753358 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:47.753651 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:47.754706 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:47.754712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:47.754723 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:47.754742 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:47.754791 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:47.754816 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:47.754973 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:47.755084 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 11:47:47.755322 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":354,"threads_started":1}
I20260924 11:47:47.757107 6553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:47.758404 6553 log.cc:824] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:47.760938 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665: No bootstrap required, opened a new log
I20260924 11:47:47.767836 6553 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1015fe22326442e84e5796339194665" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33937 } }
I20260924 11:47:47.768162 6553 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:47.768221 6553 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1015fe22326442e84e5796339194665, State: Initialized, Role: FOLLOWER
I20260924 11:47:47.768666 6553 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1015fe22326442e84e5796339194665" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33937 } }
I20260924 11:47:47.768846 6553 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:47.768932 6553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:47.769047 6553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:47.771431 6553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1015fe22326442e84e5796339194665" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33937 } }
I20260924 11:47:47.771852 6553 leader_election.cc:302] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1015fe22326442e84e5796339194665; no voters:
I20260924 11:47:47.772320 6553 leader_election.cc:288] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:47.772410 6558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:47.773232 6558 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [term 1 LEADER]: Becoming Leader. State: Replica: d1015fe22326442e84e5796339194665, State: Running, Role: LEADER
I20260924 11:47:47.773814 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1015fe22326442e84e5796339194665" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33937 } }
I20260924 11:47:47.774349 6553 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:47.776633 6560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1015fe22326442e84e5796339194665. Latest consensus state: current_term: 1 leader_uuid: "d1015fe22326442e84e5796339194665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1015fe22326442e84e5796339194665" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33937 } } }
I20260924 11:47:47.776634 6559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1015fe22326442e84e5796339194665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1015fe22326442e84e5796339194665" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33937 } } }
I20260924 11:47:47.777223 6560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:47.777279 6559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:47.778251 6563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:47.783828 6563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:47.795831 6563 catalog_manager.cc:1347] Generated new cluster ID: a946d3819fca4c6fa276b40dc1109b2a
I20260924 11:47:47.795962 6563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:47.811120 6563 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:47.811563 6563 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:47.835892 6563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665: Generated new TSK 0
I20260924 11:47:47.836947 6563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:47:47.855072 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33937
--builtin_ntp_servers=127.1.210.20:43141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 11:47:48.182651 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:48.182997 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:48.183153 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:48.192854 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:48.192982 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:48.193172 6577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:48.202286 6577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:43141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:48.203608 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:48.205343 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:48.218205 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:48.218114 6585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:48.218137 6583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:48.224911 6577 server_base.cc:1034] running on GCE node
I20260924 11:47:48.225618 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:48.226516 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:48.227705 6577 hybrid_clock.cc:648] HybridClock initialized: now 1790250468227664 us; error 32 us; skew 500 ppm
I20260924 11:47:48.228015 6577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:48.230362 6577 webserver.cc:460] Webserver started at http://127.1.210.1:38183/ using document root <none> and password file <none>
I20260924 11:47:48.231029 6577 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:48.231144 6577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:48.231483 6577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:48.234337 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "7a282e0d01094566a76bc0ec36301534"
format_stamp: "Formatted at 2026-09-24 11:47:48 on dist-test-slave-dd32"
server_key: "0e7e49fb3a1ab2bad41cd97f201c475a"
server_key_iv: "02bff695dd7fd3d8fed69ee9bca5cb4d"
server_key_version: "encryptionkey@0"
I20260924 11:47:48.235100 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "7a282e0d01094566a76bc0ec36301534"
format_stamp: "Formatted at 2026-09-24 11:47:48 on dist-test-slave-dd32"
server_key: "0e7e49fb3a1ab2bad41cd97f201c475a"
server_key_iv: "02bff695dd7fd3d8fed69ee9bca5cb4d"
server_key_version: "encryptionkey@0"
I20260924 11:47:48.239974 6577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:47:48.243153 6591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:48.244388 6577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:47:48.244557 6577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "7a282e0d01094566a76bc0ec36301534"
format_stamp: "Formatted at 2026-09-24 11:47:48 on dist-test-slave-dd32"
server_key: "0e7e49fb3a1ab2bad41cd97f201c475a"
server_key_iv: "02bff695dd7fd3d8fed69ee9bca5cb4d"
server_key_version: "encryptionkey@0"
I20260924 11:47:48.244777 6577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:48.250029 6577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:48.250754 6577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:48.251011 6577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:48.251848 6577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:48.253178 6577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:48.253284 6577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:48.253412 6577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:48.253513 6577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:48.285777 6577 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:45523
I20260924 11:47:48.285846 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:45523 every 8 connection(s)
I20260924 11:47:48.287060 6577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:48.294695 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6577
I20260924 11:47:48.294826 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:48.289498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:48405 (local address 127.1.210.62:33937)
0924 11:47:48.289917 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:48.289923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:48.291091 (+ 1168us) server_negotiation.cc:407] Connection header received
0924 11:47:48.292613 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:48.292617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:48.292684 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:48.292777 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:48.294084 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:48.294088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:48.294091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:48.294107 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:48.294145 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:48.294156 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:48.294276 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:48.294445 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 11:47:48.294511 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260924 11:47:48.294888 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:48.295503 1864 external_mini_cluster.cc:1428] Setting key 245463d110309890fe36f3550a366d70
I20260924 11:47:48.295570 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:48.290104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33937 (local address 127.1.210.1:48405)
0924 11:47:48.290943 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:48.290981 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:48.292397 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:48.292975 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:48.292982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:48.293162 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:48.293815 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:48.293818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:48.293853 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:48.293929 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:48.294247 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:48.294253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:48.294259 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:47:48.294465 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 11:47:48.294638 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":264,"threads_started":1}
I20260924 11:47:48.297180 6704 heartbeater.cc:347] Connected to a master server at 127.1.210.62:33937
I20260924 11:47:48.297446 6704 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:48.298049 6704 heartbeater.cc:511] Master 127.1.210.62:33937 requested a full tablet report, sending...
I20260924 11:47:48.298709 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33937
--builtin_ntp_servers=127.1.210.20:43141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 11:47:48.300097 6517 ts_manager.cc:194] Registered new tserver with Master: 7a282e0d01094566a76bc0ec36301534 (127.1.210.1:45523)
W20260924 11:47:48.608733 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:48.609012 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:48.609220 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:48.616330 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:48.616415 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:48.616575 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:48.625497 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:43141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:48.626684 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:48.628427 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:48.641446 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:48.641521 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:48.641517 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:48.642953 6708 server_base.cc:1034] running on GCE node
I20260924 11:47:48.643461 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:48.644266 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:48.645450 6708 hybrid_clock.cc:648] HybridClock initialized: now 1790250468645405 us; error 43 us; skew 500 ppm
I20260924 11:47:48.645746 6708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:48.648113 6708 webserver.cc:460] Webserver started at http://127.1.210.2:43353/ using document root <none> and password file <none>
I20260924 11:47:48.648831 6708 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:48.648957 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:48.649310 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:48.651989 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "870c19a50e574bdeb3d22467844ac0de"
format_stamp: "Formatted at 2026-09-24 11:47:48 on dist-test-slave-dd32"
server_key: "18b8f54c58ea0efc30ccb0968c3a4154"
server_key_iv: "b5b888409b003df3bae463e727daa105"
server_key_version: "encryptionkey@0"
I20260924 11:47:48.652724 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "870c19a50e574bdeb3d22467844ac0de"
format_stamp: "Formatted at 2026-09-24 11:47:48 on dist-test-slave-dd32"
server_key: "18b8f54c58ea0efc30ccb0968c3a4154"
server_key_iv: "b5b888409b003df3bae463e727daa105"
server_key_version: "encryptionkey@0"
I20260924 11:47:48.657747 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:47:48.660904 6722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:48.662175 6708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:47:48.662344 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "870c19a50e574bdeb3d22467844ac0de"
format_stamp: "Formatted at 2026-09-24 11:47:48 on dist-test-slave-dd32"
server_key: "18b8f54c58ea0efc30ccb0968c3a4154"
server_key_iv: "b5b888409b003df3bae463e727daa105"
server_key_version: "encryptionkey@0"
I20260924 11:47:48.662564 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:48.668258 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:48.669006 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:48.669288 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:48.670145 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:48.671414 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:48.671535 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:48.671664 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:48.671747 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:48.703570 6708 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:42457
I20260924 11:47:48.703619 6834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:42457 every 8 connection(s)
I20260924 11:47:48.704888 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:48.706370 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6708
I20260924 11:47:48.706532 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:48.707079 1864 external_mini_cluster.cc:1428] Setting key 3292df6672c024d61ae69abca6106b7e
I20260924 11:47:48.709636 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33937
--builtin_ntp_servers=127.1.210.20:43141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 11:47:48.713491 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:48.707498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:54651 (local address 127.1.210.62:33937)
0924 11:47:48.707680 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:48.707685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:48.709413 (+ 1728us) server_negotiation.cc:407] Connection header received
0924 11:47:48.711191 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:48.711197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:48.711264 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:48.711398 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:48.712729 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:48.712735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:48.712740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:48.712759 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:48.712801 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:48.712813 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:48.712936 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:48.713144 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 11:47:48.713209 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:47:48.714726 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:48.708144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33937 (local address 127.1.210.2:54651)
0924 11:47:48.709261 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:48.709304 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:48.710920 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:48.711640 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:48.711647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:48.711740 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:48.712434 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:48.712436 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:48.712473 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:48.712556 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:48.712907 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:48.712917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:48.712924 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:47:48.713227 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 11:47:48.713481 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":368,"threads_started":1}
I20260924 11:47:48.716038 6835 heartbeater.cc:347] Connected to a master server at 127.1.210.62:33937
I20260924 11:47:48.716346 6835 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:48.716948 6835 heartbeater.cc:511] Master 127.1.210.62:33937 requested a full tablet report, sending...
I20260924 11:47:48.718369 6517 ts_manager.cc:194] Registered new tserver with Master: 870c19a50e574bdeb3d22467844ac0de (127.1.210.2:42457)
W20260924 11:47:49.033105 6838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:49.033380 6838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:49.033567 6838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:49.040642 6838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:49.040728 6838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:49.040896 6838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:49.049921 6838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:43141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:49.051111 6838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:49.052814 6838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:49.065711 6843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:49.065747 6846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:49.065785 6844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:49.074620 6838 server_base.cc:1034] running on GCE node
I20260924 11:47:49.075248 6838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:49.076205 6838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:49.077399 6838 hybrid_clock.cc:648] HybridClock initialized: now 1790250469077361 us; error 33 us; skew 500 ppm
I20260924 11:47:49.077713 6838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:49.079986 6838 webserver.cc:460] Webserver started at http://127.1.210.3:44707/ using document root <none> and password file <none>
I20260924 11:47:49.080665 6838 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:49.080777 6838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:49.081141 6838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:49.083899 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "5ca6f6e656994ae2aa20d22d2d165f27"
format_stamp: "Formatted at 2026-09-24 11:47:49 on dist-test-slave-dd32"
server_key: "2bad23a8f1d2f348827f85bd8c1a1137"
server_key_iv: "1b43a2836b46f0582cf1918d92027d3a"
server_key_version: "encryptionkey@0"
I20260924 11:47:49.084653 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "5ca6f6e656994ae2aa20d22d2d165f27"
format_stamp: "Formatted at 2026-09-24 11:47:49 on dist-test-slave-dd32"
server_key: "2bad23a8f1d2f348827f85bd8c1a1137"
server_key_iv: "1b43a2836b46f0582cf1918d92027d3a"
server_key_version: "encryptionkey@0"
I20260924 11:47:49.089493 6838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 11:47:49.092649 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:49.093859 6838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:47:49.094025 6838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "5ca6f6e656994ae2aa20d22d2d165f27"
format_stamp: "Formatted at 2026-09-24 11:47:49 on dist-test-slave-dd32"
server_key: "2bad23a8f1d2f348827f85bd8c1a1137"
server_key_iv: "1b43a2836b46f0582cf1918d92027d3a"
server_key_version: "encryptionkey@0"
I20260924 11:47:49.094250 6838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:49.100039 6838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:49.100771 6838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:49.101054 6838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:49.101986 6838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:49.103257 6838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:49.103386 6838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:49.103523 6838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:49.103611 6838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:49.136689 6838 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:34537
I20260924 11:47:49.136737 6964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:34537 every 8 connection(s)
I20260924 11:47:49.138096 6838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:49.138515 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 6838
I20260924 11:47:49.138682 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:49.139284 1864 external_mini_cluster.cc:1428] Setting key 01870982dbf8d962a855af97a6303b1d
I20260924 11:47:49.146641 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.141045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:36977 (local address 127.1.210.62:33937)
0924 11:47:49.141292 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.141297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.142851 (+ 1554us) server_negotiation.cc:407] Connection header received
0924 11:47:49.144484 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.144489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.144553 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.144666 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.145938 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.145943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.145948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.145966 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.146003 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.146014 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.146134 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.146321 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.146385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 11:47:49.147585 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.141693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:33937 (local address 127.1.210.3:36977)
0924 11:47:49.142678 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.142743 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:49.144229 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:49.144890 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:49.144898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:49.144988 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:49.145663 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:49.145665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:49.145700 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:49.145781 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:49.146110 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:49.146122 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:49.146129 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:47:49.146419 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 11:47:49.146664 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":362,"threads_started":1}
I20260924 11:47:49.149005 6965 heartbeater.cc:347] Connected to a master server at 127.1.210.62:33937
I20260924 11:47:49.149294 6965 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:49.149787 6965 heartbeater.cc:511] Master 127.1.210.62:33937 requested a full tablet report, sending...
I20260924 11:47:49.151058 6517 ts_manager.cc:194] Registered new tserver with Master: 5ca6f6e656994ae2aa20d22d2d165f27 (127.1.210.3:34537)
I20260924 11:47:49.154273 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:47:49.160749 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.158436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.1.210.62:33937)
0924 11:47:49.158581 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.158585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.159086 (+ 501us) server_negotiation.cc:407] Connection header received
0924 11:47:49.159217 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.159221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.159266 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.159350 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.160209 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.160213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.160216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.160229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.160256 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.160266 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.160363 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.160514 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.160556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:47:49.164541 6517 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:47:49.172019 6517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:47:49.192579 6978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.184328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:34537 (local address 127.0.0.1:52010)
0924 11:47:49.185934 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.185950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:49.186028 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:49.189924 (+ 3896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:49.189933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:49.190028 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:49.190878 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:49.190881 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:49.190918 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:49.191021 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:49.191583 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:49.191596 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:49.191606 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:47:49.192071 (+ 465us) client_negotiation.cc:240] Negotiation successful
0924 11:47:49.192161 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":1344,"threads_started":1}
I20260924 11:47:49.192798 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.184874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.1.210.3:34537)
0924 11:47:49.189518 (+ 4644us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.189523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.189549 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:47:49.189601 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.189606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.189744 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.189916 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.191230 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.191239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.191253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.191271 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.191343 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.191365 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.191522 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.192298 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.192396 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4500,"thread_start_us":4343,"threads_started":1}
I20260924 11:47:49.193224 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.184307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.1.210.1:45523)
0924 11:47:49.186054 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.186066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.186104 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:47:49.186192 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.186203 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.186513 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.186698 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.192317 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.192339 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.192351 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.192370 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.192422 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.192442 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.192540 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.192804 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.192905 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":260,"threads_started":1}
I20260924 11:47:49.194051 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.184167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52640)
0924 11:47:49.185381 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.185416 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:49.185545 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:49.191299 (+ 5754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:49.191307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:49.191353 (+ 46us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:49.192083 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:49.192086 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:49.192103 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:49.192140 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:49.192634 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:49.192641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:49.192648 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:47:49.193534 (+ 886us) client_negotiation.cc:240] Negotiation successful
0924 11:47:49.193571 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":776,"threads_started":1}
I20260924 11:47:49.196249 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.186004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:42457 (local address 127.0.0.1:57884)
0924 11:47:49.191283 (+ 5279us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.191300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:49.192119 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:49.193684 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:49.193691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:49.193718 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:49.195045 (+ 1327us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:49.195051 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:49.195070 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:49.195112 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:49.195623 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:49.195627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:49.195632 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:47:49.195763 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 11:47:49.195809 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5170,"thread_start_us":5046,"threads_started":1}
I20260924 11:47:49.196944 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.186619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.1.210.2:42457)
0924 11:47:49.193223 (+ 6604us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.193242 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.193271 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:47:49.193333 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.193342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.193524 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.193670 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.195276 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.195284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.195295 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.195320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.195372 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.195390 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.195566 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.196519 (+ 953us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.196604 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6400,"thread_start_us":5237,"threads_started":1}
I20260924 11:47:49.196964 6900 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac332a82c684260ba7a4a9977ec7ba4 (DEFAULT_TABLE table=test-table [id=77adb14be8f44a89994f351d03005f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:49.197367 6639 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac332a82c684260ba7a4a9977ec7ba4 (DEFAULT_TABLE table=test-table [id=77adb14be8f44a89994f351d03005f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:49.198598 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac332a82c684260ba7a4a9977ec7ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:49.199865 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac332a82c684260ba7a4a9977ec7ba4 (DEFAULT_TABLE table=test-table [id=77adb14be8f44a89994f351d03005f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:47:49.200905 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac332a82c684260ba7a4a9977ec7ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:49.200912 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac332a82c684260ba7a4a9977ec7ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:49.209206 6983 tablet_bootstrap.cc:492] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: Bootstrap starting.
I20260924 11:47:49.212154 6983 tablet_bootstrap.cc:654] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:49.213253 6983 log.cc:824] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:49.215931 6986 tablet_bootstrap.cc:492] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Bootstrap starting.
I20260924 11:47:49.216676 6985 tablet_bootstrap.cc:492] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Bootstrap starting.
I20260924 11:47:49.217131 6983 tablet_bootstrap.cc:492] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: No bootstrap required, opened a new log
I20260924 11:47:49.217428 6983 ts_tablet_manager.cc:1397] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260924 11:47:49.219343 6985 tablet_bootstrap.cc:654] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:49.219491 6986 tablet_bootstrap.cc:654] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:49.220168 6985 log.cc:824] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:49.220585 6986 log.cc:824] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:49.222926 6985 tablet_bootstrap.cc:492] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: No bootstrap required, opened a new log
I20260924 11:47:49.223184 6985 ts_tablet_manager.cc:1397] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:47:49.223419 6986 tablet_bootstrap.cc:492] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: No bootstrap required, opened a new log
I20260924 11:47:49.223642 6986 ts_tablet_manager.cc:1397] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:47:49.227460 6983 raft_consensus.cc:348] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.227907 6983 raft_consensus.cc:374] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:49.228048 6983 raft_consensus.cc:729] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a282e0d01094566a76bc0ec36301534, State: Initialized, Role: FOLLOWER
I20260924 11:47:49.228660 6983 consensus_queue.cc:260] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.229773 6704 heartbeater.cc:503] Master 127.1.210.62:33937 was elected leader, sending a full tablet report...
I20260924 11:47:49.230419 6983 ts_tablet_manager.cc:1428] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260924 11:47:49.231808 6985 raft_consensus.cc:348] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.232209 6985 raft_consensus.cc:374] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:49.232366 6985 raft_consensus.cc:729] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870c19a50e574bdeb3d22467844ac0de, State: Initialized, Role: FOLLOWER
I20260924 11:47:49.233054 6985 consensus_queue.cc:260] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.233501 6986 raft_consensus.cc:348] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.233959 6986 raft_consensus.cc:374] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:49.234068 6986 raft_consensus.cc:729] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca6f6e656994ae2aa20d22d2d165f27, State: Initialized, Role: FOLLOWER
I20260924 11:47:49.234629 6986 consensus_queue.cc:260] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.235189 6835 heartbeater.cc:503] Master 127.1.210.62:33937 was elected leader, sending a full tablet report...
I20260924 11:47:49.235919 6985 ts_tablet_manager.cc:1428] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
I20260924 11:47:49.236673 6965 heartbeater.cc:503] Master 127.1.210.62:33937 was elected leader, sending a full tablet report...
I20260924 11:47:49.237303 6986 ts_tablet_manager.cc:1428] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260924 11:47:49.289366 6705 tablet.cc:2369] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:47:49.337211 6991 raft_consensus.cc:482] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:47:49.337517 6991 raft_consensus.cc:504] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.339131 6991 leader_election.cc:288] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a282e0d01094566a76bc0ec36301534 (127.1.210.1:45523), 870c19a50e574bdeb3d22467844ac0de (127.1.210.2:42457)
I20260924 11:47:49.341889 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.339497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.1.210.3:54073)
0924 11:47:49.339834 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.339858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:49.340000 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:49.340365 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:49.340369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:49.340388 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:49.341135 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:49.341137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:49.341150 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:49.341184 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:49.341484 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:49.341489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:49.341493 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 11:47:49.341594 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:47:49.341642 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260924 11:47:49.341905 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.339518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:54073 (local address 127.1.210.1:45523)
0924 11:47:49.339703 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.339709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.339961 (+ 252us) server_negotiation.cc:407] Connection header received
0924 11:47:49.340141 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.340147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.340209 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.340320 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.341319 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.341324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.341329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.341344 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.341371 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.341382 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.341510 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.341587 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.341644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:47:49.342308 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.339588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:50325 (local address 127.1.210.2:42457)
0924 11:47:49.339736 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.339742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.340468 (+ 726us) server_negotiation.cc:407] Connection header received
0924 11:47:49.340599 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.340607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.340671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.340754 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.341705 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.341711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.341715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.341733 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.341768 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.341780 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.341919 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.341998 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.342056 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:47:49.342505 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.339508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:42457 (local address 127.1.210.3:50325)
0924 11:47:49.340360 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.340377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:49.340481 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:49.340802 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:49.340807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:49.340829 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:49.341516 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:47:49.341518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:49.341531 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:47:49.341563 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:49.341891 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:49.341897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:49.341903 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:47:49.342020 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 11:47:49.342065 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":50,"thread_start_us":429,"threads_started":1}
I20260924 11:47:49.343003 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac332a82c684260ba7a4a9977ec7ba4" candidate_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a282e0d01094566a76bc0ec36301534" is_pre_election: true
I20260924 11:47:49.343477 6659 raft_consensus.cc:2454] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ca6f6e656994ae2aa20d22d2d165f27 in term 0.
I20260924 11:47:49.343433 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac332a82c684260ba7a4a9977ec7ba4" candidate_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870c19a50e574bdeb3d22467844ac0de" is_pre_election: true
I20260924 11:47:49.343847 6790 raft_consensus.cc:2454] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ca6f6e656994ae2aa20d22d2d165f27 in term 0.
I20260924 11:47:49.344115 6853 leader_election.cc:302] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ca6f6e656994ae2aa20d22d2d165f27, 7a282e0d01094566a76bc0ec36301534; no voters:
I20260924 11:47:49.344488 6991 raft_consensus.cc:2781] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:47:49.344622 6991 raft_consensus.cc:482] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:47:49.344719 6991 raft_consensus.cc:3037] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:49.346971 6991 raft_consensus.cc:504] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.347546 6991 leader_election.cc:288] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [CANDIDATE]: Term 1 election: Requested vote from peers 7a282e0d01094566a76bc0ec36301534 (127.1.210.1:45523), 870c19a50e574bdeb3d22467844ac0de (127.1.210.2:42457)
I20260924 11:47:49.347894 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac332a82c684260ba7a4a9977ec7ba4" candidate_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a282e0d01094566a76bc0ec36301534"
I20260924 11:47:49.347970 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac332a82c684260ba7a4a9977ec7ba4" candidate_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870c19a50e574bdeb3d22467844ac0de"
I20260924 11:47:49.348120 6659 raft_consensus.cc:3037] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:49.348171 6790 raft_consensus.cc:3037] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:49.350425 6790 raft_consensus.cc:2454] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ca6f6e656994ae2aa20d22d2d165f27 in term 1.
I20260924 11:47:49.350457 6659 raft_consensus.cc:2454] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ca6f6e656994ae2aa20d22d2d165f27 in term 1.
I20260924 11:47:49.350777 6853 leader_election.cc:302] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ca6f6e656994ae2aa20d22d2d165f27, 7a282e0d01094566a76bc0ec36301534; no voters:
I20260924 11:47:49.351050 6991 raft_consensus.cc:2781] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:49.351476 6991 raft_consensus.cc:686] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca6f6e656994ae2aa20d22d2d165f27, State: Running, Role: LEADER
I20260924 11:47:49.351912 6991 consensus_queue.cc:237] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } }
I20260924 11:47:49.356227 6518 catalog_manager.cc:5495] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ca6f6e656994ae2aa20d22d2d165f27 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ca6f6e656994ae2aa20d22d2d165f27" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 } health_report { overall_health: UNKNOWN } } }
I20260924 11:47:49.380046 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.378171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.1.210.3:34537)
0924 11:47:49.378378 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.378382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.378398 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:47:49.378567 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.378570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.378640 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.378719 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.379512 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.379516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.379519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.379532 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.379560 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.379571 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.379666 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.379750 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.379793 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:47:49.387619 6790 raft_consensus.cc:1261] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 1 FOLLOWER]: Refusing update from remote peer 5ca6f6e656994ae2aa20d22d2d165f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:49.387622 6659 raft_consensus.cc:1261] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 1 FOLLOWER]: Refusing update from remote peer 5ca6f6e656994ae2aa20d22d2d165f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:47:49.388545 6991 consensus_queue.cc:1035] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a282e0d01094566a76bc0ec36301534" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:47:49.388823 6993 consensus_queue.cc:1035] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870c19a50e574bdeb3d22467844ac0de" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:47:49.389822 6966 tablet.cc:2369] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:47:49.401125 6998 mvcc.cc:205] Tried to move back new op lower bound from 7332865922602815488 to 7332865922469236736. Current Snapshot: MvccSnapshot[applied={T|T < 7332865922602815488}]
I20260924 11:47:49.404310 6997 mvcc.cc:205] Tried to move back new op lower bound from 7332865922602815488 to 7332865922469236736. Current Snapshot: MvccSnapshot[applied={T|T < 7332865922602815488}]
I20260924 11:47:49.413189 6518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:49.413444 6518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:47:49.416817 6518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1015fe22326442e84e5796339194665: Sending DeleteTablet for 3 replicas of tablet 4ac332a82c684260ba7a4a9977ec7ba4
I20260924 11:47:49.417927 6639 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac332a82c684260ba7a4a9977ec7ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:49 UTC) from {username='slave'} at 127.0.0.1:52640
I20260924 11:47:49.417984 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac332a82c684260ba7a4a9977ec7ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:49 UTC) from {username='slave'} at 127.0.0.1:52010
I20260924 11:47:49.418058 6770 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac332a82c684260ba7a4a9977ec7ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:47:49 UTC) from {username='slave'} at 127.0.0.1:57884
I20260924 11:47:49.420398 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6577
I20260924 11:47:49.425558 7007 tablet_replica.cc:330] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: stopping tablet replica
I20260924 11:47:49.425750 7006 tablet_replica.cc:330] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534: stopping tablet replica
I20260924 11:47:49.426007 7007 raft_consensus.cc:2229] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:47:49.426313 7008 tablet_replica.cc:330] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: stopping tablet replica
I20260924 11:47:49.426613 7007 raft_consensus.cc:2258] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:49.426728 7008 raft_consensus.cc:2229] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:49.427250 7006 raft_consensus.cc:2229] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:47:49.427397 7008 raft_consensus.cc:2258] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:49.429437 7008 ts_tablet_manager.cc:1905] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:49.429446 7007 ts_tablet_manager.cc:1905] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:47:49.436239 7008 ts_tablet_manager.cc:1918] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:49.436403 7008 log.cc:1196] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/4ac332a82c684260ba7a4a9977ec7ba4
I20260924 11:47:49.429050 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.421730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.1.210.1:45523)
0924 11:47:49.421957 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.421966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.426488 (+ 4522us) server_negotiation.cc:407] Connection header received
0924 11:47:49.426706 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.426711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.426806 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.426919 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.428237 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.428243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.428248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.428266 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.428301 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.428316 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.428407 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.428693 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.428756 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:47:49.436436 7007 ts_tablet_manager.cc:1918] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:47:49.436578 7007 log.cc:1196] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/4ac332a82c684260ba7a4a9977ec7ba4
I20260924 11:47:49.437150 7007 ts_tablet_manager.cc:1939] T 4ac332a82c684260ba7a4a9977ec7ba4 P 5ca6f6e656994ae2aa20d22d2d165f27: Deleting consensus metadata
I20260924 11:47:49.437165 6699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:49.437181 7008 ts_tablet_manager.cc:1939] T 4ac332a82c684260ba7a4a9977ec7ba4 P 870c19a50e574bdeb3d22467844ac0de: Deleting consensus metadata
I20260924 11:47:49.438829 6504 catalog_manager.cc:4841] TS 870c19a50e574bdeb3d22467844ac0de (127.1.210.2:42457): tablet 4ac332a82c684260ba7a4a9977ec7ba4 (table test-table [id=77adb14be8f44a89994f351d03005f5d]) successfully deleted
I20260924 11:47:49.439160 6504 catalog_manager.cc:4841] TS 5ca6f6e656994ae2aa20d22d2d165f27 (127.1.210.3:34537): tablet 4ac332a82c684260ba7a4a9977ec7ba4 (table test-table [id=77adb14be8f44a89994f351d03005f5d]) successfully deleted
I20260924 11:47:49.535132 7006 raft_consensus.cc:2258] T 4ac332a82c684260ba7a4a9977ec7ba4 P 7a282e0d01094566a76bc0ec36301534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:47:49.535480 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6577
I20260924 11:47:49.552320 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6708
W20260924 11:47:49.552309 6502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.1:45523 (error 108)
I20260924 11:47:49.554293 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.553757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52654)
0924 11:47:49.553981 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.554124 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260924 11:47:49.554584 6502 catalog_manager.cc:4584] TS 7a282e0d01094566a76bc0ec36301534 (127.1.210.1:45523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ac332a82c684260ba7a4a9977ec7ba4: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
I20260924 11:47:49.555869 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.553702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.1.210.2:42457)
0924 11:47:49.553895 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.553902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.553923 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:47:49.553982 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.553989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.554112 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.554218 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.555227 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.555231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.555234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.555250 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.555278 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.555289 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.555390 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.555511 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.555556 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:47:49.556486 6830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:49.608273 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.607660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52656)
0924 11:47:49.607926 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.608077 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 11:47:49.609426 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.609036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52670)
0924 11:47:49.609213 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.609295 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:47:49.643388 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6708
I20260924 11:47:49.660030 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6838
I20260924 11:47:49.663166 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.660834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.1.210.3:34537)
0924 11:47:49.661042 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:49.661049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:49.661176 (+ 127us) server_negotiation.cc:407] Connection header received
0924 11:47:49.661440 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:49.661444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:49.661494 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:49.661615 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:49.662517 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:49.662521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:49.662524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:49.662539 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:49.662571 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:49.662585 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:49.662727 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:49.662823 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 11:47:49.662881 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:47:49.663816 6960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:49.669907 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.669442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52674)
0924 11:47:49.669636 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.669729 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:47:49.670869 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.670499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52676)
0924 11:47:49.670646 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.670728 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:47:49.751266 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.750845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52678)
0924 11:47:49.751050 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.751135 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 11:47:49.752318 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:49.751941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:45523 (local address 127.0.0.1:52680)
0924 11:47:49.752143 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:49.752204 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 11:47:49.781780 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6838
I20260924 11:47:49.800289 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 6487
I20260924 11:47:49.801378 6548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:49.903951 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 6487
2026-09-24T11:47:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2615 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 11:47:49.944077 1864 mini_oidc.cc:152] Starting JWKS server
I20260924 11:47:49.946539 1864 webserver.cc:460] Webserver started at https://127.0.0.1:45581/ using document root <none> and password file <none>
I20260924 11:47:49.946803 1864 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:47:49.947413 1864 webserver.cc:460] Webserver started at http://127.0.0.1:45135/ using document root <none> and password file <none>
2026-09-24T11:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:49Z Disabled control of system clock
I20260924 11:47:49.972048 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:38433
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:38433
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/testchainca.cert with env {}
W20260924 11:47:50.285364 7020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:50.285627 7020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:50.285709 7020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:50.292622 7020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:50.292711 7020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:50.292747 7020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:50.292799 7020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:50.292866 7020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:47:50.292919 7020 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:47:50.292958 7020 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45581/jwks.json
I20260924 11:47:50.302397 7020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:38433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:38433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:50.303524 7020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:50.304716 7020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:50.316015 7025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:50.315984 7026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:50.316051 7028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:50.316757 7020 server_base.cc:1034] running on GCE node
I20260924 11:47:50.317500 7020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:50.318753 7020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:50.319942 7020 hybrid_clock.cc:648] HybridClock initialized: now 1790250470319904 us; error 38 us; skew 500 ppm
I20260924 11:47:50.320282 7020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:50.322443 7020 webserver.cc:460] Webserver started at http://127.1.210.62:46117/ using document root <none> and password file <none>
I20260924 11:47:50.323076 7020 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:50.323192 7020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:50.323503 7020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:50.326311 7020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "2638505589eb412a80733c0874175f7f"
format_stamp: "Formatted at 2026-09-24 11:47:50 on dist-test-slave-dd32"
server_key: "50fea51eb2337a0ab21a5886bd6a7a0d"
server_key_iv: "cb5aa16f84c6f9726cf447c93017ba41"
server_key_version: "encryptionkey@0"
I20260924 11:47:50.327075 7020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "2638505589eb412a80733c0874175f7f"
format_stamp: "Formatted at 2026-09-24 11:47:50 on dist-test-slave-dd32"
server_key: "50fea51eb2337a0ab21a5886bd6a7a0d"
server_key_iv: "cb5aa16f84c6f9726cf447c93017ba41"
server_key_version: "encryptionkey@0"
I20260924 11:47:50.331861 7020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:47:50.335211 7034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:50.336545 7020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:50.336725 7020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "2638505589eb412a80733c0874175f7f"
format_stamp: "Formatted at 2026-09-24 11:47:50 on dist-test-slave-dd32"
server_key: "50fea51eb2337a0ab21a5886bd6a7a0d"
server_key_iv: "cb5aa16f84c6f9726cf447c93017ba41"
server_key_version: "encryptionkey@0"
I20260924 11:47:50.336937 7020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:50.347383 7020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:50.348117 7020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:50.348426 7020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:50.366065 7020 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:38433
I20260924 11:47:50.366102 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:38433 every 8 connection(s)
I20260924 11:47:50.367303 7020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:50.370826 7086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:50.372426 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7020
I20260924 11:47:50.372640 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:50.373457 1864 external_mini_cluster.cc:1428] Setting key 7ad48f3498195020983072ac97405027
I20260924 11:47:50.384328 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f: Bootstrap starting.
I20260924 11:47:50.386332 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:50.375211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.1.210.62:38433)
0924 11:47:50.376169 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:50.376179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:50.376219 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:47:50.377818 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:50.377841 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:50.378266 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:50.378555 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:50.379127 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:50.380338 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:50.382222 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:50.382819 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:50.382939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:50.384127 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:50.384134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:50.384146 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:50.384698 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:50.384767 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:50.384786 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:50.384873 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:50.385135 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 11:47:50.385440 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":368,"threads_started":1}
I20260924 11:47:50.387440 7086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:50.388409 7086 log.cc:824] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:50.389979 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:47:50.391485 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f: No bootstrap required, opened a new log
I20260924 11:47:50.399098 7086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638505589eb412a80733c0874175f7f" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38433 } }
I20260924 11:47:50.399408 7086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:50.399477 7086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2638505589eb412a80733c0874175f7f, State: Initialized, Role: FOLLOWER
I20260924 11:47:50.400010 7086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638505589eb412a80733c0874175f7f" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38433 } }
I20260924 11:47:50.400198 7086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:50.400292 7086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:50.400413 7086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:50.402474 7086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638505589eb412a80733c0874175f7f" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38433 } }
I20260924 11:47:50.402894 7086 leader_election.cc:302] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2638505589eb412a80733c0874175f7f; no voters:
I20260924 11:47:50.403519 7086 leader_election.cc:288] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:50.403648 7091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:50.404253 7091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [term 1 LEADER]: Becoming Leader. State: Replica: 2638505589eb412a80733c0874175f7f, State: Running, Role: LEADER
I20260924 11:47:50.404891 7091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638505589eb412a80733c0874175f7f" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38433 } }
I20260924 11:47:50.405954 7086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:50.407708 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2638505589eb412a80733c0874175f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638505589eb412a80733c0874175f7f" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38433 } } }
I20260924 11:47:50.407698 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2638505589eb412a80733c0874175f7f. Latest consensus state: current_term: 1 leader_uuid: "2638505589eb412a80733c0874175f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638505589eb412a80733c0874175f7f" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 38433 } } }
I20260924 11:47:50.408181 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:50.408221 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:50.410171 7096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:50.415200 7096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:50.422930 7096 catalog_manager.cc:1347] Generated new cluster ID: 517d545303c84797a9bf404e47dece80
I20260924 11:47:50.423033 7096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:50.432477 7096 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:50.434196 7096 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:50.444999 7096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2638505589eb412a80733c0874175f7f: Generated new TSK 0
I20260924 11:47:50.445988 7096 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 11:47:50.491974 7116 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 11:47:50.492146 7116 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:47:50.493187 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:50.472028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.1.210.62:38433)
0924 11:47:50.472220 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:50.472227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:50.472801 (+ 574us) server_negotiation.cc:407] Connection header received
0924 11:47:50.473082 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:50.473088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:50.473183 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:50.473337 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:47:50.473802 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:50.474895 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:50.476220 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:50.476902 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:50.477018 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:50.477550 (+ 532us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:47:50.492537 (+ 14987us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 11:47:50.492876 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 11:47:50.493307 7089 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 11:47:50.494761 7115 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:50.472017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:38433 (local address 127.0.0.1:35420)
0924 11:47:50.472702 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:50.472717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:50.472858 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:50.473368 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:50.473371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:50.473382 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:47:50.473668 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:50.473676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:50.475016 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:50.475019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:50.476092 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:50.476102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:50.477053 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:50.477057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:50.477242 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:50.477295 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:47:50.492801 (+ 15506us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 11:47:50.493030 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.62:38433: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":362,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.210.62:38433: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 11:47:50.496665 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7020
I20260924 11:47:50.497761 7081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:50.587898 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7020
2026-09-24T11:47:50Z chronyd exiting
I20260924 11:47:50.637441 1864 test_util.cc:175] -----------------------------------------------
I20260924 11:47:50.637598 1864 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790250429340952-1864-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (697 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 11:47:50.641196 1864 mini_oidc.cc:152] Starting JWKS server
I20260924 11:47:50.642982 1864 webserver.cc:460] Webserver started at https://127.0.0.1:33491/ using document root <none> and password file <none>
I20260924 11:47:50.643213 1864 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:47:50.643810 1864 webserver.cc:460] Webserver started at http://127.0.0.1:45661/ using document root <none> and password file <none>
2026-09-24T11:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:50Z Disabled control of system clock
I20260924 11:47:50.668354 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:45925
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:45925
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/testchainca.cert with env {}
W20260924 11:47:50.979501 7123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:50.979765 7123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:50.979849 7123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:50.986724 7123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:50.986822 7123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:50.986867 7123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:50.986922 7123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:50.986964 7123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:47:50.987016 7123 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:47:50.987056 7123 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33491/jwks.json
I20260924 11:47:50.996980 7123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:45925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:50.998172 7123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:50.999387 7123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:51.010841 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:51.010874 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:51.010926 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:51.011433 7123 server_base.cc:1034] running on GCE node
I20260924 11:47:51.012243 7123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:51.013594 7123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:51.014807 7123 hybrid_clock.cc:648] HybridClock initialized: now 1790250471014774 us; error 23 us; skew 500 ppm
I20260924 11:47:51.015100 7123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:51.017204 7123 webserver.cc:460] Webserver started at http://127.1.210.62:36685/ using document root <none> and password file <none>
I20260924 11:47:51.017820 7123 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:51.017905 7123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:51.018174 7123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:51.020825 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "312ed13aec14427a88e18e146153b023"
format_stamp: "Formatted at 2026-09-24 11:47:51 on dist-test-slave-dd32"
server_key: "3880c22f6d83a51ba92bdc3ee9a9b2b4"
server_key_iv: "549e19d15a14e222b4aedf7275027263"
server_key_version: "encryptionkey@0"
I20260924 11:47:51.021608 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "312ed13aec14427a88e18e146153b023"
format_stamp: "Formatted at 2026-09-24 11:47:51 on dist-test-slave-dd32"
server_key: "3880c22f6d83a51ba92bdc3ee9a9b2b4"
server_key_iv: "549e19d15a14e222b4aedf7275027263"
server_key_version: "encryptionkey@0"
I20260924 11:47:51.026244 7123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:47:51.029404 7137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:51.030670 7123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:47:51.030802 7123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "312ed13aec14427a88e18e146153b023"
format_stamp: "Formatted at 2026-09-24 11:47:51 on dist-test-slave-dd32"
server_key: "3880c22f6d83a51ba92bdc3ee9a9b2b4"
server_key_iv: "549e19d15a14e222b4aedf7275027263"
server_key_version: "encryptionkey@0"
I20260924 11:47:51.030958 7123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:51.050638 7123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:51.051318 7123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:51.051558 7123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:51.068687 7123 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:45925
I20260924 11:47:51.068708 7188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:45925 every 8 connection(s)
I20260924 11:47:51.069995 7123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:51.073695 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:51.075264 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7123
I20260924 11:47:51.075448 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:51.076267 1864 external_mini_cluster.cc:1428] Setting key 12aae80547a98f318301f614c383989e
I20260924 11:47:51.087131 7189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023: Bootstrap starting.
I20260924 11:47:51.089259 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:51.078079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.1.210.62:45925)
0924 11:47:51.079041 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:51.079055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:51.079098 (+ 43us) server_negotiation.cc:407] Connection header received
0924 11:47:51.080558 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:51.080581 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:51.080999 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:51.081356 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:51.081838 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.083088 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.084920 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.085609 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.085767 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:51.086988 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:51.086994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:51.087007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:51.087577 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:51.087648 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:51.087668 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:51.087753 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:51.087984 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 11:47:51.088305 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":370,"threads_started":1}
I20260924 11:47:51.090305 7189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:51.091298 7189 log.cc:824] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:51.092839 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:47:51.094432 7189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023: No bootstrap required, opened a new log
I20260924 11:47:51.101892 7189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ed13aec14427a88e18e146153b023" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45925 } }
I20260924 11:47:51.102203 7189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:51.102275 7189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312ed13aec14427a88e18e146153b023, State: Initialized, Role: FOLLOWER
I20260924 11:47:51.102800 7189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ed13aec14427a88e18e146153b023" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45925 } }
I20260924 11:47:51.102993 7189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:51.103086 7189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:51.103199 7189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:51.106829 7189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ed13aec14427a88e18e146153b023" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45925 } }
I20260924 11:47:51.107256 7189 leader_election.cc:302] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 312ed13aec14427a88e18e146153b023; no voters:
I20260924 11:47:51.107780 7189 leader_election.cc:288] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:51.107925 7194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:51.108592 7194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [term 1 LEADER]: Becoming Leader. State: Replica: 312ed13aec14427a88e18e146153b023, State: Running, Role: LEADER
I20260924 11:47:51.109282 7194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ed13aec14427a88e18e146153b023" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45925 } }
I20260924 11:47:51.110000 7189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:51.112293 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 312ed13aec14427a88e18e146153b023. Latest consensus state: current_term: 1 leader_uuid: "312ed13aec14427a88e18e146153b023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ed13aec14427a88e18e146153b023" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45925 } } }
I20260924 11:47:51.112203 7195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "312ed13aec14427a88e18e146153b023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ed13aec14427a88e18e146153b023" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45925 } } }
I20260924 11:47:51.112613 7195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:51.112613 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:51.114648 7202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:51.119556 7202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:51.127810 7202 catalog_manager.cc:1347] Generated new cluster ID: 825e0a81e2f7470097a1a5d5f9cec275
I20260924 11:47:51.127933 7202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:51.146099 7202 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:51.147928 7202 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:51.158658 7202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 312ed13aec14427a88e18e146153b023: Generated new TSK 0
I20260924 11:47:51.159538 7202 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 11:47:51.201932 7219 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 11:47:51.202059 7219 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:47:51.203258 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:51.183315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.1.210.62:45925)
0924 11:47:51.183507 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:51.183514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:51.184019 (+ 505us) server_negotiation.cc:407] Connection header received
0924 11:47:51.184226 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:51.184232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:51.184324 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:51.184488 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:47:51.184915 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.186078 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.187430 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.188132 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.188249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:51.188757 (+ 508us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:47:51.202594 (+ 13837us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 11:47:51.202947 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45938: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 11:47:51.203281 7218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:51.183260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45925 (local address 127.0.0.1:45938)
0924 11:47:51.183922 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:51.183937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:51.184053 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:51.184513 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:51.184516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:51.184526 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:47:51.184756 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:51.184764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.186217 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.186221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:51.187292 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:51.187303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.188304 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.188308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:51.188463 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:51.188494 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:47:51.202800 (+ 14306us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 11:47:51.202995 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.62:45925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":371,"threads_started":1}
W20260924 11:47:51.203380 7192 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45938: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 11:47:51.204639 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7123
I20260924 11:47:51.205849 7184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:51.299822 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7123
2026-09-24T11:47:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 11:47:51.354429 1864 mini_oidc.cc:152] Starting JWKS server
I20260924 11:47:51.356225 1864 webserver.cc:460] Webserver started at https://127.0.0.1:37255/ using document root <none> and password file <none>
I20260924 11:47:51.356422 1864 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:47:51.356994 1864 webserver.cc:460] Webserver started at http://127.0.0.1:45525/ using document root <none> and password file <none>
2026-09-24T11:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:51Z Disabled control of system clock
I20260924 11:47:51.381549 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42087
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:42087
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37255/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 11:47:51.696460 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:51.696830 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:51.696993 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:51.704366 7226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:51.704527 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:51.704627 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:51.704756 7226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:51.704846 7226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:47:51.704946 7226 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:47:51.705029 7226 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37255/jwks.json
I20260924 11:47:51.714301 7226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:42087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37255/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:51.715668 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:51.717633 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:51.729734 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:51.729702 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:51.730047 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:51.730798 7226 server_base.cc:1034] running on GCE node
I20260924 11:47:51.731565 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:51.733001 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:51.734241 7226 hybrid_clock.cc:648] HybridClock initialized: now 1790250471734209 us; error 36 us; skew 500 ppm
I20260924 11:47:51.734596 7226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:51.739787 7226 webserver.cc:460] Webserver started at http://127.1.210.62:46587/ using document root <none> and password file <none>
I20260924 11:47:51.740562 7226 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:51.740700 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:51.741010 7226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:51.743758 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "1ad53ff0d92f43bcba4f080ac32e673d"
format_stamp: "Formatted at 2026-09-24 11:47:51 on dist-test-slave-dd32"
server_key: "0e40b4a580c42629a4cdf769ba9d9ad0"
server_key_iv: "f338474411919e8bbb0a1812d231499d"
server_key_version: "encryptionkey@0"
I20260924 11:47:51.744514 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "1ad53ff0d92f43bcba4f080ac32e673d"
format_stamp: "Formatted at 2026-09-24 11:47:51 on dist-test-slave-dd32"
server_key: "0e40b4a580c42629a4cdf769ba9d9ad0"
server_key_iv: "f338474411919e8bbb0a1812d231499d"
server_key_version: "encryptionkey@0"
I20260924 11:47:51.749468 7226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 11:47:51.752614 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:51.753868 7226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:47:51.754043 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "1ad53ff0d92f43bcba4f080ac32e673d"
format_stamp: "Formatted at 2026-09-24 11:47:51 on dist-test-slave-dd32"
server_key: "0e40b4a580c42629a4cdf769ba9d9ad0"
server_key_iv: "f338474411919e8bbb0a1812d231499d"
server_key_version: "encryptionkey@0"
I20260924 11:47:51.754248 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:51.769301 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:51.770035 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:51.770324 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:51.787814 7226 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:42087
I20260924 11:47:51.787822 7291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:42087 every 8 connection(s)
I20260924 11:47:51.789165 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:51.792816 7292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:51.797554 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7226
I20260924 11:47:51.797773 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:51.798395 1864 external_mini_cluster.cc:1428] Setting key 246a9e8faaee0c038ee7dd4390b7b0fa
I20260924 11:47:51.803481 7292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d: Bootstrap starting.
I20260924 11:47:51.806458 7292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:51.807435 7292 log.cc:824] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:51.810495 7292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d: No bootstrap required, opened a new log
I20260924 11:47:51.811290 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:51.800234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.1.210.62:42087)
0924 11:47:51.801252 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:51.801264 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:51.801303 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:47:51.802902 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:51.802923 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:51.803350 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:51.803728 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:51.804250 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.805634 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.807001 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.807716 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.807913 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:51.809145 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:51.809153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:51.809167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:51.809658 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:51.809727 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:51.809746 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:51.809837 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:51.810071 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 11:47:51.810355 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":333,"threads_started":1}
I20260924 11:47:51.815037 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:47:51.818981 7292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42087 } }
I20260924 11:47:51.819322 7292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:51.819443 7292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad53ff0d92f43bcba4f080ac32e673d, State: Initialized, Role: FOLLOWER
I20260924 11:47:51.819967 7292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42087 } }
I20260924 11:47:51.820186 7292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:51.820322 7292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:51.820516 7292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:51.823119 7292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42087 } }
I20260924 11:47:51.823601 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad53ff0d92f43bcba4f080ac32e673d; no voters:
I20260924 11:47:51.824152 7292 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:51.824290 7297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:51.825016 7297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [term 1 LEADER]: Becoming Leader. State: Replica: 1ad53ff0d92f43bcba4f080ac32e673d, State: Running, Role: LEADER
I20260924 11:47:51.825567 7297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42087 } }
I20260924 11:47:51.826352 7292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:51.828392 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42087 } } }
I20260924 11:47:51.828810 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:51.829348 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad53ff0d92f43bcba4f080ac32e673d. Latest consensus state: current_term: 1 leader_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad53ff0d92f43bcba4f080ac32e673d" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42087 } } }
I20260924 11:47:51.829686 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:51.833611 7306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:51.837623 7306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:51.845831 7306 catalog_manager.cc:1347] Generated new cluster ID: 44a291115ffc45e498d06229a694f24d
I20260924 11:47:51.845940 7306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:51.869539 7306 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:51.870786 7306 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:51.878245 7306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad53ff0d92f43bcba4f080ac32e673d: Generated new TSK 0
I20260924 11:47:51.879009 7306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:47:51.908577 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:51.900248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.1.210.62:42087)
0924 11:47:51.900453 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:51.900460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:51.900760 (+ 300us) server_negotiation.cc:407] Connection header received
0924 11:47:51.900943 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:51.900950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:51.901049 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:51.901268 (+ 219us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:47:51.901679 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.902827 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.904142 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.904868 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.905017 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:51.905519 (+ 502us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:47:51.907877 (+ 2358us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 11:47:51.908229 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 11:47:51.908717 7295 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 11:47:51.910269 7321 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:51.900085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42087 (local address 127.0.0.1:59968)
0924 11:47:51.900661 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:51.900679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:51.900789 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:51.901269 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:51.901272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:51.901282 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:47:51.901539 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:51.901547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.902954 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.902958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:51.904006 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:51.904016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:51.905051 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:51.905055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:51.905229 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:51.905259 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:47:51.908094 (+ 2835us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 11:47:51.909853 (+ 1759us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.62:42087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":1462,"thread_start_us":312,"threads_started":1}
I20260924 11:47:51.911738 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7226
I20260924 11:47:51.912992 7287 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 11:47:51.920101 7322 webserver.cc:533] Webserver: sslize failed:
E20260924 11:47:51.920184 7322 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:47:52.004592 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7226
2026-09-24T11:47:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (701 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T11:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:52Z Disabled control of system clock
I20260924 11:47:52.080578 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:44207
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:44207
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/testchainca.cert with env {}
W20260924 11:47:52.391705 7327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:52.392001 7327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:52.392086 7327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:52.399058 7327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:52.399147 7327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:52.399181 7327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:52.399235 7327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:52.399268 7327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:47:52.399318 7327 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:47:52.399348 7327 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 11:47:52.409340 7327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:44207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:52.410486 7327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:52.411656 7327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:52.423141 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:52.423101 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:52.423333 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:52.423810 7327 server_base.cc:1034] running on GCE node
I20260924 11:47:52.424546 7327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:52.425978 7327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:52.427210 7327 hybrid_clock.cc:648] HybridClock initialized: now 1790250472427178 us; error 46 us; skew 500 ppm
I20260924 11:47:52.427548 7327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:47:52.429834 7327 webserver.cc:460] Webserver started at http://127.1.210.62:34083/ using document root <none> and password file <none>
I20260924 11:47:52.430521 7327 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:52.430651 7327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:52.430968 7327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:52.433739 7327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "adb4932d55304823976c385b7ffe0a95"
format_stamp: "Formatted at 2026-09-24 11:47:52 on dist-test-slave-dd32"
server_key: "c5b2ce4d69d4543ae52a428f4d3d4bd9"
server_key_iv: "7e46358209add60d8607d15f3a02613f"
server_key_version: "encryptionkey@0"
I20260924 11:47:52.434523 7327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "adb4932d55304823976c385b7ffe0a95"
format_stamp: "Formatted at 2026-09-24 11:47:52 on dist-test-slave-dd32"
server_key: "c5b2ce4d69d4543ae52a428f4d3d4bd9"
server_key_iv: "7e46358209add60d8607d15f3a02613f"
server_key_version: "encryptionkey@0"
I20260924 11:47:52.439390 7327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:47:52.442701 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:52.443985 7327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:47:52.444164 7327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "adb4932d55304823976c385b7ffe0a95"
format_stamp: "Formatted at 2026-09-24 11:47:52 on dist-test-slave-dd32"
server_key: "c5b2ce4d69d4543ae52a428f4d3d4bd9"
server_key_iv: "7e46358209add60d8607d15f3a02613f"
server_key_version: "encryptionkey@0"
I20260924 11:47:52.444367 7327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:52.464557 7327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:52.465421 7327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:52.465704 7327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:52.483055 7327 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:44207
I20260924 11:47:52.483074 7392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:44207 every 8 connection(s)
I20260924 11:47:52.484323 7327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:52.486874 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7327
I20260924 11:47:52.487061 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:52.487825 1864 external_mini_cluster.cc:1428] Setting key ef98e46743fe7e10cf0068a5671761f3
I20260924 11:47:52.488499 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:52.501314 7396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:52.489678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.1.210.62:44207)
0924 11:47:52.490578 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:52.490587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:52.490625 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:47:52.492682 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:52.492701 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:52.493225 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:52.493545 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:52.494092 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:52.495362 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:52.497123 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:52.497734 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:52.497867 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:52.499106 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:52.499114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:52.499126 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:47:52.499649 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:52.499723 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:47:52.499747 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:52.499831 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:52.500054 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 11:47:52.500367 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":328,"threads_started":1}
I20260924 11:47:52.501623 7393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95: Bootstrap starting.
I20260924 11:47:52.504678 7393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:52.504762 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:47:52.505865 7393 log.cc:824] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:52.508539 7393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95: No bootstrap required, opened a new log
I20260924 11:47:52.515764 7393 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb4932d55304823976c385b7ffe0a95" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44207 } }
I20260924 11:47:52.516160 7393 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:52.516283 7393 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb4932d55304823976c385b7ffe0a95, State: Initialized, Role: FOLLOWER
I20260924 11:47:52.516816 7393 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb4932d55304823976c385b7ffe0a95" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44207 } }
I20260924 11:47:52.517040 7393 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:52.517220 7393 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:52.517400 7393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:52.520085 7393 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb4932d55304823976c385b7ffe0a95" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44207 } }
I20260924 11:47:52.520594 7393 leader_election.cc:302] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb4932d55304823976c385b7ffe0a95; no voters:
I20260924 11:47:52.521229 7393 leader_election.cc:288] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:52.521332 7398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:52.522051 7398 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [term 1 LEADER]: Becoming Leader. State: Replica: adb4932d55304823976c385b7ffe0a95, State: Running, Role: LEADER
I20260924 11:47:52.522563 7398 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb4932d55304823976c385b7ffe0a95" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44207 } }
I20260924 11:47:52.523315 7393 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:52.526078 7400 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb4932d55304823976c385b7ffe0a95. Latest consensus state: current_term: 1 leader_uuid: "adb4932d55304823976c385b7ffe0a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb4932d55304823976c385b7ffe0a95" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44207 } } }
I20260924 11:47:52.526052 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb4932d55304823976c385b7ffe0a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb4932d55304823976c385b7ffe0a95" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44207 } } }
I20260924 11:47:52.526465 7400 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:52.526521 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:52.528692 7404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:52.534606 7404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:52.542645 7404 catalog_manager.cc:1347] Generated new cluster ID: ab93a1b42f1e4293ae0aff1293fce41c
I20260924 11:47:52.542771 7404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:52.553251 7404 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:52.554428 7404 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:52.573417 7404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb4932d55304823976c385b7ffe0a95: Generated new TSK 0
I20260924 11:47:52.574255 7404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:47:52.636256 7396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:52.593569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.1.210.62:44207)
0924 11:47:52.593763 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:52.593769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:52.594298 (+ 529us) server_negotiation.cc:407] Connection header received
0924 11:47:52.594516 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:52.594519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:52.594589 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:52.594713 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:47:52.595245 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:52.595972 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:52.598039 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:52.598535 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:52.598648 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:52.599189 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:47:52.635731 (+ 36542us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 11:47:52.636029 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 11:47:52.636430 7396 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 11:47:52.636706 7422 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:52.593447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:44207 (local address 127.0.0.1:39194)
0924 11:47:52.594168 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:52.594187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:52.594343 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:52.594764 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:52.594769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:52.594782 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:47:52.595101 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:52.595113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:52.596103 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:52.596108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:52.597890 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:52.597902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:52.598709 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:52.598715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:52.598896 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:52.598938 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:47:52.636002 (+ 37064us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 11:47:52.636297 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.62:44207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":428,"threads_started":1}
I20260924 11:47:52.637923 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7327
I20260924 11:47:52.639008 7388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:52.733474 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7327
2026-09-24T11:47:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (711 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (417 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (404 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:53 dist-test-slave-dd32 krb5kdc[7437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:53 dist-test-slave-dd32 krb5kdc[7437](info): set up 2 sockets
Sep 24 11:47:53 dist-test-slave-dd32 krb5kdc[7437](info): commencing operation
krb5kdc: starting...
W20260924 11:47:54.705660 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:54 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:47:54.900266 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36219
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:55.214602 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:55.214926 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:55.215090 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:55.222139 7450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:55.222272 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:55.222386 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:55.222500 7450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:55.222577 7450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:55.231949 7450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:55.233234 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:55.234483 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:55.246276 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:55.246228 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:55.246330 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:55.247040 7450 server_base.cc:1034] running on GCE node
I20260924 11:47:55.247865 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:55.249256 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:55.250450 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790250475250391 us; error 51 us; skew 500 ppm
I20260924 11:47:55.250761 7450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:55 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:55.253731 7450 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:47:55.255062 7450 webserver.cc:460] Webserver started at http://127.1.210.62:40769/ using document root <none> and password file <none>
I20260924 11:47:55.255731 7450 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:55.255860 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:55.256198 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:55.259054 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "670d06f3af8b4b729a3a8ffd199f4349"
format_stamp: "Formatted at 2026-09-24 11:47:55 on dist-test-slave-dd32"
server_key: "c8032cfab56b4d02faf339b9795e8d08"
server_key_iv: "2c516171f312c030cc369dfb32a82300"
server_key_version: "encryptionkey@0"
I20260924 11:47:55.259784 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "670d06f3af8b4b729a3a8ffd199f4349"
format_stamp: "Formatted at 2026-09-24 11:47:55 on dist-test-slave-dd32"
server_key: "c8032cfab56b4d02faf339b9795e8d08"
server_key_iv: "2c516171f312c030cc369dfb32a82300"
server_key_version: "encryptionkey@0"
I20260924 11:47:55.264552 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260924 11:47:55.267867 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:55.269114 7450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:55.269289 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "670d06f3af8b4b729a3a8ffd199f4349"
format_stamp: "Formatted at 2026-09-24 11:47:55 on dist-test-slave-dd32"
server_key: "c8032cfab56b4d02faf339b9795e8d08"
server_key_iv: "2c516171f312c030cc369dfb32a82300"
server_key_version: "encryptionkey@0"
I20260924 11:47:55.269483 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:55.287909 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:55.291661 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:55.291926 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:55.308378 7450 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:36219
I20260924 11:47:55.308409 7516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:36219 every 8 connection(s)
I20260924 11:47:55.309731 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:55.313301 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:55.318106 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7450
I20260924 11:47:55.318315 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:55.318910 1864 external_mini_cluster.cc:1428] Setting key e22906d09f416728d0d913935374a722
I20260924 11:47:55.323618 7517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349: Bootstrap starting.
I20260924 11:47:55.327230 7517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:55.328406 7517 log.cc:824] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349: Log is configured to *not* fsync() on all Append() calls
I20260924 11:47:55.331105 7517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349: No bootstrap required, opened a new log
Sep 24 11:47:55 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:55.338409 7517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670d06f3af8b4b729a3a8ffd199f4349" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36219 } }
I20260924 11:47:55.338754 7517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:55.338871 7517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670d06f3af8b4b729a3a8ffd199f4349, State: Initialized, Role: FOLLOWER
I20260924 11:47:55.339574 7517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670d06f3af8b4b729a3a8ffd199f4349" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36219 } }
I20260924 11:47:55.339833 7517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:55.339995 7517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:55.340196 7517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:55.340757 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:55.320770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.1.210.62:36219)
0924 11:47:55.321659 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:55.321670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:55.321714 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:47:55.323311 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:55.323331 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:55.323691 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:55.324049 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:55.325773 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:55.327012 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:55.328731 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:55.329465 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:55.329667 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:55.333518 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:55.333525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:55.333539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:55.333575 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:55.336601 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:55.337385 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:55.337394 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:55.337407 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:55.337581 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:55.338010 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:55.338015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:55.338020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:55.338509 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:55.338832 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:55.339226 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 11:47:55.339609 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":449,"threads_started":1}
I20260924 11:47:55.342991 7517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670d06f3af8b4b729a3a8ffd199f4349" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36219 } }
I20260924 11:47:55.343519 7517 leader_election.cc:302] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 670d06f3af8b4b729a3a8ffd199f4349; no voters:
I20260924 11:47:55.344113 7517 leader_election.cc:288] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:55.344269 7522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:55.344949 7522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [term 1 LEADER]: Becoming Leader. State: Replica: 670d06f3af8b4b729a3a8ffd199f4349, State: Running, Role: LEADER
I20260924 11:47:55.345527 7522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670d06f3af8b4b729a3a8ffd199f4349" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36219 } }
I20260924 11:47:55.346110 7517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:55.348595 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 670d06f3af8b4b729a3a8ffd199f4349. Latest consensus state: current_term: 1 leader_uuid: "670d06f3af8b4b729a3a8ffd199f4349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670d06f3af8b4b729a3a8ffd199f4349" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36219 } } }
I20260924 11:47:55.348584 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "670d06f3af8b4b729a3a8ffd199f4349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670d06f3af8b4b729a3a8ffd199f4349" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36219 } } }
I20260924 11:47:55.348963 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:55.348970 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:55.350492 7528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:55.355031 7528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:55.362910 7528 catalog_manager.cc:1347] Generated new cluster ID: 9fec5b59f77341dda4e3c8c00bae38f1
I20260924 11:47:55.363024 7528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:55.364114 7539 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:47:55.379074 7528 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:55.380321 7528 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:55.386000 7528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 670d06f3af8b4b729a3a8ffd199f4349: Generated new TSK 0
I20260924 11:47:55.386761 7528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:47:55.438674 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36219
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:55.757670 7543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:55.757923 7543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:55.758085 7543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:55.765009 7543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:55.765120 7543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:55.765326 7543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:55.774928 7543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:55.776100 7543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:55.777328 7543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:55.790055 7548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:55.790084 7549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:55.790022 7551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:55.790782 7543 server_base.cc:1034] running on GCE node
I20260924 11:47:55.791431 7543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:55.792341 7543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:55.793561 7543 hybrid_clock.cc:648] HybridClock initialized: now 1790250475793519 us; error 56 us; skew 500 ppm
I20260924 11:47:55.793871 7543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:55 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:55.797225 7543 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:47:55.798687 7543 webserver.cc:460] Webserver started at http://127.1.210.1:33123/ using document root <none> and password file <none>
I20260924 11:47:55.799381 7543 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:55.799528 7543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:55.799834 7543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:55.802739 7543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "6a67dd219e5445a6a2d70bba053abace"
format_stamp: "Formatted at 2026-09-24 11:47:55 on dist-test-slave-dd32"
server_key: "b205353708ac11d434e47d69e0f90173"
server_key_iv: "e50abca7b21b93c64d8203907df19727"
server_key_version: "encryptionkey@0"
I20260924 11:47:55.803515 7543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "6a67dd219e5445a6a2d70bba053abace"
format_stamp: "Formatted at 2026-09-24 11:47:55 on dist-test-slave-dd32"
server_key: "b205353708ac11d434e47d69e0f90173"
server_key_iv: "e50abca7b21b93c64d8203907df19727"
server_key_version: "encryptionkey@0"
I20260924 11:47:55.808377 7543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:47:55.811713 7558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:55.812938 7543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:47:55.813146 7543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "6a67dd219e5445a6a2d70bba053abace"
format_stamp: "Formatted at 2026-09-24 11:47:55 on dist-test-slave-dd32"
server_key: "b205353708ac11d434e47d69e0f90173"
server_key_iv: "e50abca7b21b93c64d8203907df19727"
server_key_version: "encryptionkey@0"
I20260924 11:47:55.813340 7543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:55.834121 7543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:55.838280 7543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:55.838564 7543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:55.839490 7543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:55.841383 7543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:55.841487 7543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:55.841588 7543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:55.841632 7543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:55.875591 7543 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:36837
I20260924 11:47:55.875625 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:36837 every 8 connection(s)
I20260924 11:47:55.876935 7543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:55.878854 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7543
I20260924 11:47:55.879068 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:55.879714 1864 external_mini_cluster.cc:1428] Setting key 982f1f1d22863bfe1ece5743cad32b59
Sep 24 11:47:55 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:47:55.905210 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:55.882988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:53923 (local address 127.1.210.62:36219)
0924 11:47:55.888230 (+ 5242us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:55.888234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:55.888259 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:47:55.888331 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:55.888335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:55.888391 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:55.888526 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:55.890223 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:55.890950 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:55.893685 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:55.894068 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:55.894181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:55.898411 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:55.898416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:55.898419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:55.898474 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:55.902020 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:55.902727 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:55.902736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:55.902742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:55.902925 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:55.903345 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:55.903352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:55.903358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:55.903788 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:55.903964 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:55.904658 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 11:47:55.904836 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5147,"thread_start_us":5073,"threads_started":1}
I20260924 11:47:55.906131 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:55.882390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36219 (local address 127.1.210.1:53923)
0924 11:47:55.883242 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:55.883306 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:55.884980 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:55.888775 (+ 3795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:55.888788 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:55.889393 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:55.890041 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:55.890066 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:55.891117 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:55.891125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:55.893515 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:55.893545 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:55.894319 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:55.894329 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:55.894670 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:55.895411 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:55.895447 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:55.898197 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:55.902158 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:55.902164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:55.902177 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:55.902582 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:55.903046 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:55.903049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:55.903051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:55.903210 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:55.903914 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:55.903920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:55.904224 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 11:47:55.904496 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 11:47:55.904863 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":366,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:55.907729 7671 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36219
I20260924 11:47:55.908051 7671 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:55.908898 7671 heartbeater.cc:511] Master 127.1.210.62:36219 requested a full tablet report, sending...
I20260924 11:47:55.911054 7482 ts_manager.cc:194] Registered new tserver with Master: 6a67dd219e5445a6a2d70bba053abace (127.1.210.1:36837)
I20260924 11:47:55.912464 7482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:53923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:47:55.932248 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36219
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:56.241693 7677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:56.241994 7677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:56.242163 7677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:56.249274 7677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:56.249410 7677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:56.249624 7677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:47:56.259002 7677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:56.260082 7677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:56.261313 7677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:56.274075 7683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:56.274021 7682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:56.274111 7685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:56.274860 7677 server_base.cc:1034] running on GCE node
I20260924 11:47:56.275561 7677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:56.276697 7677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:56.277972 7677 hybrid_clock.cc:648] HybridClock initialized: now 1790250476277872 us; error 64 us; skew 500 ppm
I20260924 11:47:56.278368 7677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.282562 7677 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:47:56.284173 7677 webserver.cc:460] Webserver started at http://127.1.210.2:41549/ using document root <none> and password file <none>
I20260924 11:47:56.284817 7677 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:56.284966 7677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:56.285319 7677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:56.288057 7677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "9c4bc2eed2f04c0d9bf0a530a5413585"
format_stamp: "Formatted at 2026-09-24 11:47:56 on dist-test-slave-dd32"
server_key: "351c087a6ac0599fd334fb8240094f9e"
server_key_iv: "7f38dcd9a0ef41011c1e8fac75a67790"
server_key_version: "encryptionkey@0"
I20260924 11:47:56.288805 7677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "9c4bc2eed2f04c0d9bf0a530a5413585"
format_stamp: "Formatted at 2026-09-24 11:47:56 on dist-test-slave-dd32"
server_key: "351c087a6ac0599fd334fb8240094f9e"
server_key_iv: "7f38dcd9a0ef41011c1e8fac75a67790"
server_key_version: "encryptionkey@0"
I20260924 11:47:56.293612 7677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:47:56.296876 7692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:56.298142 7677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:56.298312 7677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "9c4bc2eed2f04c0d9bf0a530a5413585"
format_stamp: "Formatted at 2026-09-24 11:47:56 on dist-test-slave-dd32"
server_key: "351c087a6ac0599fd334fb8240094f9e"
server_key_iv: "7f38dcd9a0ef41011c1e8fac75a67790"
server_key_version: "encryptionkey@0"
I20260924 11:47:56.298511 7677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:56.313493 7677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:56.317561 7677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:56.317900 7677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:56.318845 7677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:56.320158 7677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:56.320257 7677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:56.320400 7677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:56.320474 7677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:56.353334 7677 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39973
I20260924 11:47:56.353379 7804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39973 every 8 connection(s)
I20260924 11:47:56.354602 7677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:47:56.360662 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7677
I20260924 11:47:56.360828 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:47:56.361347 1864 external_mini_cluster.cc:1428] Setting key 1f36225040ea73b5f91ed1a86a2365b4
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:56.375291 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.356983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:41729 (local address 127.1.210.62:36219)
0924 11:47:56.357180 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.357184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.358298 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 11:47:56.360036 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.360041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.360139 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.360247 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.361866 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.362714 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.364988 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.365443 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.365552 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.369679 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:56.369683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:56.369687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:56.369716 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:56.372305 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.372992 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.373003 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.373006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.373208 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.373663 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.373668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.373672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.373921 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:56.374096 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:56.374727 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 11:47:56.374932 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:47:56.376212 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.357465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36219 (local address 127.1.210.2:41729)
0924 11:47:56.358150 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:56.358185 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:56.359773 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:56.360475 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:56.360482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:56.361100 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:56.361708 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:56.361724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.362870 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.362875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:56.364828 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:56.364845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.365674 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.365683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:56.366015 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.366702 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:56.366733 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:56.369438 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:56.372438 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:56.372447 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:56.372471 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:56.372856 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:56.373342 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:56.373347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:56.373351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:56.373544 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:56.374057 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:56.374065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:56.374382 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 11:47:56.374646 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 11:47:56.375139 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":272,"threads_started":1}
I20260924 11:47:56.377712 7805 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36219
I20260924 11:47:56.378051 7805 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:56.378798 7805 heartbeater.cc:511] Master 127.1.210.62:36219 requested a full tablet report, sending...
I20260924 11:47:56.380191 7482 ts_manager.cc:194] Registered new tserver with Master: 9c4bc2eed2f04c0d9bf0a530a5413585 (127.1.210.2:39973)
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:47:56.381004 7482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:41729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:47:56.404259 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36219
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:56.718933 7810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:56.719190 7810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:56.719280 7810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:56.726269 7810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:56.726361 7810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:56.726465 7810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:47:56.735810 7810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:56.736938 7810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:56.738178 7810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:56.751453 7815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:56.751511 7816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:56.751672 7818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:56.752564 7810 server_base.cc:1034] running on GCE node
I20260924 11:47:56.753139 7810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:56.753968 7810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:56.755157 7810 hybrid_clock.cc:648] HybridClock initialized: now 1790250476755116 us; error 37 us; skew 500 ppm
I20260924 11:47:56.755455 7810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.764089 7810 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:47:56.765544 7810 webserver.cc:460] Webserver started at http://127.1.210.3:38463/ using document root <none> and password file <none>
I20260924 11:47:56.766201 7810 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:56.766326 7810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:56.766623 7810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:56.769550 7810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "ada81effddf24b3a809c81889d0654b4"
format_stamp: "Formatted at 2026-09-24 11:47:56 on dist-test-slave-dd32"
server_key: "8010699fe9bfce8f277872fb851a273f"
server_key_iv: "b80e1f0fb1ad25019114dc962332e23a"
server_key_version: "encryptionkey@0"
I20260924 11:47:56.770334 7810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "ada81effddf24b3a809c81889d0654b4"
format_stamp: "Formatted at 2026-09-24 11:47:56 on dist-test-slave-dd32"
server_key: "8010699fe9bfce8f277872fb851a273f"
server_key_iv: "b80e1f0fb1ad25019114dc962332e23a"
server_key_version: "encryptionkey@0"
I20260924 11:47:56.775436 7810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 11:47:56.778667 7825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:56.779862 7810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:47:56.780033 7810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "ada81effddf24b3a809c81889d0654b4"
format_stamp: "Formatted at 2026-09-24 11:47:56 on dist-test-slave-dd32"
server_key: "8010699fe9bfce8f277872fb851a273f"
server_key_iv: "b80e1f0fb1ad25019114dc962332e23a"
server_key_version: "encryptionkey@0"
I20260924 11:47:56.780244 7810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:56.818742 7810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:56.822767 7810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:56.823060 7810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:56.824034 7810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:56.825373 7810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:56.825445 7810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:56.825613 7810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:56.825663 7810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:56.859192 7810 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:35495
I20260924 11:47:56.859249 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:35495 every 8 connection(s)
I20260924 11:47:56.860486 7810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:47:56.862249 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7810
I20260924 11:47:56.862439 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:47:56.862869 1864 external_mini_cluster.cc:1428] Setting key aa3a43b5c395e4a50d5258d1af300d15
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:47:56.880620 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.863075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:46775 (local address 127.1.210.62:36219)
0924 11:47:56.863275 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.863282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.864690 (+ 1408us) server_negotiation.cc:407] Connection header received
0924 11:47:56.866573 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.866577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.866637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.866715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.868015 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.868729 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.870253 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.870639 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.870732 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.874177 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:56.874182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:56.874185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:56.874213 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:56.877347 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.878371 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.878377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.878380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.878523 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.878987 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.878992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.878995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.879322 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:56.879480 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:56.880101 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 11:47:56.880306 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:47:56.881368 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.863693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36219 (local address 127.1.210.3:46775)
0924 11:47:56.864534 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:56.864584 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:56.866346 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:56.866869 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:56.866877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:56.867343 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:56.867879 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:56.867896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.868879 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.868882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:56.870133 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:56.870144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.870808 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.870812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:56.871005 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.871680 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:56.871701 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:56.874013 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:56.877473 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:56.877482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:56.877496 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:56.878046 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:56.878624 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:56.878628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:56.878630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:56.878863 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:56.879444 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:56.879449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:56.879738 (+ 289us) client_negotiation.cc:769] Sending connection context
0924 11:47:56.879937 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 11:47:56.880299 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":379,"threads_started":1}
I20260924 11:47:56.882722 7938 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36219
I20260924 11:47:56.883019 7938 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:56.883752 7938 heartbeater.cc:511] Master 127.1.210.62:36219 requested a full tablet report, sending...
I20260924 11:47:56.885108 7482 ts_manager.cc:194] Registered new tserver with Master: ada81effddf24b3a809c81889d0654b4 (127.1.210.3:35495)
I20260924 11:47:56.885882 7482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:46775
I20260924 11:47:56.890276 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/test-admin.keytab.
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.915354 7671 heartbeater.cc:503] Master 127.1.210.62:36219 was elected leader, sending a full tablet report...
I20260924 11:47:56.916392 1864 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 11:47:56.921558 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:56.919115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.1.210.1:36837)
0924 11:47:56.919787 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.919793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.919816 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:47:56.919867 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.919874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.920071 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.920247 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.921121 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34524: BlockingRecv error: recv got EOF from 127.0.0.1:34524 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":334,"threads_started":1}
W20260924 11:47:56.921590 1864 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.923025 1864 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.924327 1864 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:56.942723 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.925896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.1.210.1:36837)
0924 11:47:56.926045 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.926049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.926546 (+ 497us) server_negotiation.cc:407] Connection header received
0924 11:47:56.926667 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.926670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.926720 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.926798 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.928140 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.929934 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.931931 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.932806 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.932927 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.938050 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:56.938059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:56.938074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:56.938118 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:56.940467 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.941036 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.941043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.941045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.941226 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.941550 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.941553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.941554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.941865 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:56.942072 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:56.942291 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 11:47:56.942464 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 11:47:56.948315 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:56.946626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.1.210.1:36837)
0924 11:47:56.946829 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.946836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.947086 (+ 250us) server_negotiation.cc:407] Connection header received
0924 11:47:56.947261 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.947266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.947332 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.947433 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.948078 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34544: BlockingRecv error: recv got EOF from 127.0.0.1:34544 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:47:56.948684 1864 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc)
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.950172 1864 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.951594 1864 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:56.964828 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.952867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.1.210.1:36837)
0924 11:47:56.953090 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.953098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.953286 (+ 188us) server_negotiation.cc:407] Connection header received
0924 11:47:56.953478 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.953483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.953550 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.953649 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.954558 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.955247 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.956414 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.956844 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.956983 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.960132 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:56.960145 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:56.960148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:56.960176 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:56.962456 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.963072 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.963078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.963082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.963203 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.963579 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.963584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.963587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.963836 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:56.964104 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:56.964367 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 11:47:56.964555 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 11:47:56.968742 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:47:56.967311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.1.210.1:36837)
0924 11:47:56.967452 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.967458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.967676 (+ 218us) server_negotiation.cc:407] Connection header received
0924 11:47:56.967797 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.967800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.967858 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.967932 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.968492 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34558: BlockingRecv error: recv got EOF from 127.0.0.1:34558 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 11:47:56.968946 1864 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790250429340952-1864-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.970361 1864 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:56.971658 1864 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:47:56 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:47:56.985260 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.972858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.1.210.1:36837)
0924 11:47:56.973101 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.973107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.973349 (+ 242us) server_negotiation.cc:407] Connection header received
0924 11:47:56.973503 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.973506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.973554 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.973628 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.974471 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.975129 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.976264 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.976669 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.976757 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.980507 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:56.980511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:56.980515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:56.980542 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:56.982821 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.983467 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.983470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.983472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.983601 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.983951 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.983954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.983956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.984184 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:56.984432 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:56.984664 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 11:47:56.984895 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:47:56.986716 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7543
I20260924 11:47:56.997499 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:56.987391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.1.210.1:36837)
0924 11:47:56.987539 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:56.987543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:56.987865 (+ 322us) server_negotiation.cc:407] Connection header received
0924 11:47:56.987983 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:56.987986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:56.988031 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:56.988103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:56.988876 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.989593 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.990736 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:56.991135 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:56.991223 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:56.992988 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:56.992992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:56.992995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:56.993022 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:56.995475 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.996013 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.996017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.996019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.996136 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:56.996421 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:56.996424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:56.996426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:56.996645 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:56.996829 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:56.997051 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:47:56.997258 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:47:56.998106 7666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:57.079909 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7543
I20260924 11:47:57.097339 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7677
Sep 24 11:47:57 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:47:57.112605 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:57.098210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.1.210.2:39973)
0924 11:47:57.098711 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:57.098718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:57.098742 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:47:57.098801 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:57.098808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:57.099028 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:57.099201 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:57.100149 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:57.101282 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:57.102599 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:57.103215 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:57.103355 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:57.106977 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:57.106988 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:57.107002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:57.107032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:57.109620 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:57.110233 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:57.110241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:57.110243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:57.110358 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:57.110722 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:57.110729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:57.110732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:57.111190 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:57.111521 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:57.111874 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 11:47:57.112137 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
I20260924 11:47:57.114374 7800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:57.197345 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7677
I20260924 11:47:57.212491 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7810
Sep 24 11:47:57 dist-test-slave-dd32 krb5kdc[7437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:47:57.227553 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:57.213619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.1.210.3:35495)
0924 11:47:57.214254 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:57.214261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:57.214290 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:47:57.214347 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:57.214353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:57.214564 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:57.214686 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:57.215589 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:57.216592 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:57.217803 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:57.218333 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:57.218426 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:57.222197 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:57.222207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:57.222221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:57.222253 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:57.224740 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:57.225630 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:57.225637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:57.225639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:57.225752 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:57.226127 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:57.226136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:57.226138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:57.226439 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:57.226765 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:57.226997 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 11:47:57.227248 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":347,"threads_started":1}
I20260924 11:47:57.228758 7933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:57.311347 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7810
I20260924 11:47:57.330560 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7450
I20260924 11:47:57.331624 7512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:47:57.430644 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7450
2026-09-24T11:47:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3880 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:47:57 dist-test-slave-dd32 krb5kdc[7965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:47:57 dist-test-slave-dd32 krb5kdc[7965](info): set up 2 sockets
Sep 24 11:47:57 dist-test-slave-dd32 krb5kdc[7965](info): commencing operation
krb5kdc: starting...
W20260924 11:47:58.525835 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:47:58 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:47:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab.
Entry for principal oryx/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab.
I20260924 11:47:58.719949 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43409
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab
--principal=oryx/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:59.039359 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:59.039680 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:59.039860 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:59.047109 7978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:47:59.047247 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:59.047351 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:59.047461 7978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:47:59.047569 7978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:47:59.057452 7978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab
--principal=oryx/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:59.058729 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:59.060066 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:59.072134 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:59.072146 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:59.072175 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:59.073143 7978 server_base.cc:1034] running on GCE node
I20260924 11:47:59.073930 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:59.075287 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:59.076522 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790250479076466 us; error 64 us; skew 500 ppm
I20260924 11:47:59.076884 7978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:59 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250479, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:59.079809 7978 init.cc:377] Logged in from keytab as oryx/127.1.210.62@KRBTEST.COM (short username oryx)
I20260924 11:47:59.081233 7978 webserver.cc:460] Webserver started at http://127.1.210.62:38237/ using document root <none> and password file <none>
I20260924 11:47:59.081900 7978 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:59.082041 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:59.082324 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:59.085132 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "5aa51beb1207440099d8e59611229b44"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "26e1448b16679ad0966af94d3a480c8a"
server_key_iv: "49ee7ff849b60c06bfe332bddf971d8e"
server_key_version: "encryptionkey@0"
I20260924 11:47:59.085878 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "5aa51beb1207440099d8e59611229b44"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "26e1448b16679ad0966af94d3a480c8a"
server_key_iv: "49ee7ff849b60c06bfe332bddf971d8e"
server_key_version: "encryptionkey@0"
I20260924 11:47:59.090557 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:47:59.093726 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:59.094955 7978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:47:59.095121 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "5aa51beb1207440099d8e59611229b44"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "26e1448b16679ad0966af94d3a480c8a"
server_key_iv: "49ee7ff849b60c06bfe332bddf971d8e"
server_key_version: "encryptionkey@0"
I20260924 11:47:59.095302 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:59.107635 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:59.111312 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:59.111573 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:59.129501 7978 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43409
I20260924 11:47:59.129531 8044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43409 every 8 connection(s)
I20260924 11:47:59.130820 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:47:59.134428 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:47:59.138173 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 7978
I20260924 11:47:59.138375 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:47:59.139166 1864 external_mini_cluster.cc:1428] Setting key 0ccb6ea13c4db0fabc40d367106226a0
I20260924 11:47:59.147388 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Bootstrap starting.
I20260924 11:47:59.151161 8045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Neither blocks nor log segments found. Creating new log.
I20260924 11:47:59.152494 8045 log.cc:824] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:47:59 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:47:59.155495 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: No bootstrap required, opened a new log
I20260924 11:47:59.161679 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:59.141191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.1.210.62:43409)
0924 11:47:59.141927 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:59.141938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:59.141975 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:47:59.143428 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:59.143447 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:59.143811 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:59.144123 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:59.145837 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:59.147096 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:59.148787 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:59.149537 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:59.149666 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:59.154329 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:59.154337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:59.154351 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:59.154384 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:59.157541 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:59.158255 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:59.158264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:59.158269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:59.158435 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:59.158857 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:59.158868 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:59.158870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:59.159344 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:59.159698 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:59.160045 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:47:59.160451 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":355,"threads_started":1}
I20260924 11:47:59.164307 8045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } }
I20260924 11:47:59.164718 8045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:47:59.164821 8045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa51beb1207440099d8e59611229b44, State: Initialized, Role: FOLLOWER
I20260924 11:47:59.165448 8045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } }
I20260924 11:47:59.165673 8045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:47:59.165818 8045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:47:59.165989 8045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:47:59.168015 8045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } }
I20260924 11:47:59.168478 8045 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa51beb1207440099d8e59611229b44; no voters:
I20260924 11:47:59.169174 8045 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:47:59.169269 8050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:47:59.170913 8045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:47:59.175876 8050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa51beb1207440099d8e59611229b44, State: Running, Role: LEADER
I20260924 11:47:59.176354 8050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } }
I20260924 11:47:59.181336 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } }
I20260924 11:47:59.181696 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:47:59.181469 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa51beb1207440099d8e59611229b44. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } }
I20260924 11:47:59.181834 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
W20260924 11:47:59.186302 8064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:47:59.186434 8064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:47:59.186566 8065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:47:59.188189 8065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:47:59.195533 8065 catalog_manager.cc:1347] Generated new cluster ID: 45533faa9e354fc9aa415961c72bdb5e
I20260924 11:47:59.195637 8065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:47:59.213301 8065 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:47:59.215072 8065 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:47:59.222708 8065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Generated new TSK 0
I20260924 11:47:59.223652 8065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab.
Entry for principal oryx/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab.
I20260924 11:47:59.284801 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43409
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:47:59.610035 8071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:47:59.610345 8071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:47:59.610519 8071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:47:59.617952 8071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:47:59.618093 8071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:47:59.618266 8071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:47:59.627521 8071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:47:59.628788 8071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:47:59.630424 8071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:47:59.643563 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:59.643604 8077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:47:59.643667 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:47:59.644388 8071 server_base.cc:1034] running on GCE node
I20260924 11:47:59.644950 8071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:47:59.645879 8071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:47:59.647099 8071 hybrid_clock.cc:648] HybridClock initialized: now 1790250479647067 us; error 53 us; skew 500 ppm
I20260924 11:47:59.647396 8071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:47:59 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250479, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:47:59.650373 8071 init.cc:377] Logged in from keytab as oryx/127.1.210.1@KRBTEST.COM (short username oryx)
I20260924 11:47:59.651679 8071 webserver.cc:460] Webserver started at http://127.1.210.1:35837/ using document root <none> and password file <none>
I20260924 11:47:59.652331 8071 fs_manager.cc:359] Metadata directory not provided
I20260924 11:47:59.652472 8071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:47:59.652762 8071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:47:59.655644 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "4c3ce6361a0c481eaf81bc1b655b8a53"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "c8b605ca43fac96f9ac53b756e18ec3f"
server_key_iv: "34a3be5727cbbd4b2c263fde56a2df76"
server_key_version: "encryptionkey@0"
I20260924 11:47:59.656365 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "4c3ce6361a0c481eaf81bc1b655b8a53"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "c8b605ca43fac96f9ac53b756e18ec3f"
server_key_iv: "34a3be5727cbbd4b2c263fde56a2df76"
server_key_version: "encryptionkey@0"
I20260924 11:47:59.661330 8071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:47:59.664481 8086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:59.665782 8071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:47:59.665943 8071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "4c3ce6361a0c481eaf81bc1b655b8a53"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "c8b605ca43fac96f9ac53b756e18ec3f"
server_key_iv: "34a3be5727cbbd4b2c263fde56a2df76"
server_key_version: "encryptionkey@0"
I20260924 11:47:59.666139 8071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:47:59.693212 8071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:47:59.696882 8071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:47:59.697233 8071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:47:59.698154 8071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:47:59.699532 8071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:47:59.699628 8071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:59.699787 8071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:47:59.699856 8071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:47:59.732246 8071 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:44009
I20260924 11:47:59.732290 8198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:44009 every 8 connection(s)
I20260924 11:47:59.733592 8071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:47:59.734926 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8071
I20260924 11:47:59.735131 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:47:59.735936 1864 external_mini_cluster.cc:1428] Setting key e29c2fe069d0e345b0ef115f4432c615
Sep 24 11:47:59 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250479, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:47:59.752875 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:59.736289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:38315 (local address 127.1.210.62:43409)
0924 11:47:59.736729 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 11:47:59.736735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:47:59.737773 (+ 1038us) server_negotiation.cc:407] Connection header received
0924 11:47:59.739526 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:47:59.739531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:47:59.739617 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:47:59.739737 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:47:59.741125 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:59.741843 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:59.743682 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:47:59.744058 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:47:59.744173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:59.747473 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:47:59.747478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:47:59.747482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:47:59.747511 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:47:59.749993 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:59.750701 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:59.750712 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:59.750714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:59.750823 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:47:59.751238 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:47:59.751243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:47:59.751246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:47:59.751493 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:47:59.751642 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:47:59.752303 (+ 661us) server_negotiation.cc:299] Negotiation successful
0924 11:47:59.752521 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":224,"threads_started":1}
I20260924 11:47:59.753885 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:47:59.736868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:38315)
0924 11:47:59.737607 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 11:47:59.737668 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 11:47:59.739286 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:47:59.739952 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:47:59.739959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:47:59.740413 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:47:59.740949 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:59.740965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:59.741990 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:59.741996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:59.743520 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:47:59.743537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:47:59.744261 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:47:59.744265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:47:59.744489 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:47:59.745192 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:47:59.745214 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:47:59.747270 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:47:59.750141 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:59.750150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:59.750166 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:59.750574 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:59.750932 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:47:59.750937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:47:59.750940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:47:59.751119 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:47:59.751623 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:47:59.751630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:47:59.751944 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 11:47:59.752196 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 11:47:59.752695 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for oryx/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:47:59.755370 8199 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43409
I20260924 11:47:59.755674 8199 heartbeater.cc:464] Registering TS with master...
I20260924 11:47:59.756402 8199 heartbeater.cc:511] Master 127.1.210.62:43409 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.2@KRBTEST.COM" created.
I20260924 11:47:59.758589 8010 ts_manager.cc:194] Registered new tserver with Master: 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009)
I20260924 11:47:59.759992 8010 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:38315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab.
Entry for principal oryx/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab.
I20260924 11:47:59.780648 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43409
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:00.092377 8205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:00.092684 8205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:00.092779 8205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:00.099977 8205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:00.100085 8205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:00.100193 8205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:00.109647 8205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:00.110841 8205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:00.112076 8205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:00.125423 8211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:00.125530 8213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:00.125542 8210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:00.126336 8205 server_base.cc:1034] running on GCE node
I20260924 11:48:00.127038 8205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:00.127975 8205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:00.129204 8205 hybrid_clock.cc:648] HybridClock initialized: now 1790250480129136 us; error 57 us; skew 500 ppm
I20260924 11:48:00.129572 8205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:00 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250480, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:00.133289 8205 init.cc:377] Logged in from keytab as oryx/127.1.210.2@KRBTEST.COM (short username oryx)
I20260924 11:48:00.134946 8205 webserver.cc:460] Webserver started at http://127.1.210.2:38461/ using document root <none> and password file <none>
I20260924 11:48:00.135689 8205 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:00.135846 8205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:00.136171 8205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:00.139413 8205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "90f2d46a63fa478b82cead56cfaa6259"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "7abba75df46639e52f2de6e1b115c35c"
server_key_iv: "06544978fb83fc6c42819056ec96d7d3"
server_key_version: "encryptionkey@0"
I20260924 11:48:00.140250 8205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "90f2d46a63fa478b82cead56cfaa6259"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "7abba75df46639e52f2de6e1b115c35c"
server_key_iv: "06544978fb83fc6c42819056ec96d7d3"
server_key_version: "encryptionkey@0"
I20260924 11:48:00.146152 8205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:48:00.150108 8220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:00.151625 8205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 11:48:00.151834 8205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "90f2d46a63fa478b82cead56cfaa6259"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "7abba75df46639e52f2de6e1b115c35c"
server_key_iv: "06544978fb83fc6c42819056ec96d7d3"
server_key_version: "encryptionkey@0"
I20260924 11:48:00.152081 8205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:00.170651 8205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:00.175526 8205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:00.175832 8205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:00.176826 8205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:00.178326 8205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:00.178400 8205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:00.178588 8205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:00.178644 8205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:00.210906 8205 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:42945
I20260924 11:48:00.210953 8332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:42945 every 8 connection(s)
I20260924 11:48:00.212284 8205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:00.218938 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8205
I20260924 11:48:00.219163 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:48:00.219583 1864 external_mini_cluster.cc:1428] Setting key 50918d77de4c13cf0507cccb9b3fe976
Sep 24 11:48:00 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250480, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:00.233362 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:00.214856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:33471 (local address 127.1.210.62:43409)
0924 11:48:00.215031 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:00.215035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:00.216137 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 11:48:00.217994 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:00.218000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:00.218094 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:00.218200 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:00.219785 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.220726 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.222803 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.223614 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.223755 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:00.227103 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:00.227112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:00.227121 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:00.227164 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:00.230074 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:00.231033 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:00.231037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:00.231039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:00.231158 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:00.231639 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:00.231642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:00.231644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:00.231902 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:00.232005 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:00.232900 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 11:48:00.233123 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:48:00.234935 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:00.215286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:33471)
0924 11:48:00.215966 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:00.216004 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:00.217728 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:00.218430 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:00.218439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:00.218934 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:00.219587 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:00.219605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.220884 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.220889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:00.222632 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:00.222650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.223821 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.223825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:00.224028 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:00.224714 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:00.224735 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:00.226857 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:00.230223 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:00.230236 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:00.230256 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:00.230898 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:00.231267 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:00.231274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:00.231281 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:00.231528 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:00.232032 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:00.232043 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:00.232520 (+ 477us) client_negotiation.cc:769] Sending connection context
0924 11:48:00.232828 (+ 308us) client_negotiation.cc:240] Negotiation successful
0924 11:48:00.233435 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260924 11:48:00.236459 8333 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43409
I20260924 11:48:00.236773 8333 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:00.237615 8333 heartbeater.cc:511] Master 127.1.210.62:43409 requested a full tablet report, sending...
I20260924 11:48:00.239138 8010 ts_manager.cc:194] Registered new tserver with Master: 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2:42945)
I20260924 11:48:00.239924 8010 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:33471
WARNING: no policy specified for oryx/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab.
Entry for principal oryx/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab.
I20260924 11:48:00.267807 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43409
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:00.587456 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:00.587759 8338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:00.587936 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:00.595366 8338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:00.595551 8338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:00.595748 8338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:00.605448 8338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:00.606688 8338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:00.607992 8338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:00.621188 8343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:00.621325 8344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:00.621286 8346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:00.622155 8338 server_base.cc:1034] running on GCE node
I20260924 11:48:00.622725 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:00.623584 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:00.624805 8338 hybrid_clock.cc:648] HybridClock initialized: now 1790250480624768 us; error 54 us; skew 500 ppm
I20260924 11:48:00.625182 8338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:00 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250480, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:00.628183 8338 init.cc:377] Logged in from keytab as oryx/127.1.210.3@KRBTEST.COM (short username oryx)
I20260924 11:48:00.629714 8338 webserver.cc:460] Webserver started at http://127.1.210.3:35179/ using document root <none> and password file <none>
I20260924 11:48:00.630374 8338 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:00.630514 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:00.630802 8338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:00.633607 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "cfde18f86de144a59dde6f2436d80440"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "cae14bd8eab8c5769141c37b12f9e1cb"
server_key_iv: "6161ad613ba5db7936d90008a8a38b4a"
server_key_version: "encryptionkey@0"
I20260924 11:48:00.634336 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "cfde18f86de144a59dde6f2436d80440"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "cae14bd8eab8c5769141c37b12f9e1cb"
server_key_iv: "6161ad613ba5db7936d90008a8a38b4a"
server_key_version: "encryptionkey@0"
I20260924 11:48:00.639084 8338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:00.642299 8353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:00.643512 8338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:00.643675 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "cfde18f86de144a59dde6f2436d80440"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "cae14bd8eab8c5769141c37b12f9e1cb"
server_key_iv: "6161ad613ba5db7936d90008a8a38b4a"
server_key_version: "encryptionkey@0"
I20260924 11:48:00.643869 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:00.672209 8338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:00.676651 8338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:00.676942 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:00.677896 8338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:00.679188 8338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:00.679294 8338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:00.679437 8338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:00.679517 8338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:00.712581 8338 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:46459
I20260924 11:48:00.712637 8465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:46459 every 8 connection(s)
I20260924 11:48:00.714097 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:00.717043 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8338
I20260924 11:48:00.717258 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:48:00.717674 1864 external_mini_cluster.cc:1428] Setting key e0cb61f2c092ef5cbb6be95138d3cbe1
Sep 24 11:48:00 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250480, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:00.733958 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:00.716572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:47607 (local address 127.1.210.62:43409)
0924 11:48:00.716829 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:00.716837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:00.718132 (+ 1295us) server_negotiation.cc:407] Connection header received
0924 11:48:00.720356 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:00.720362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:00.720510 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:00.720636 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:00.722204 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.722932 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.724366 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.724736 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.724823 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:00.727957 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:00.727962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:00.727966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:00.727995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:00.730612 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:00.731402 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:00.731409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:00.731416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:00.731561 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:00.732085 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:00.732090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:00.732093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:00.732408 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:00.732558 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:00.733429 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 11:48:00.733633 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 11:48:00.734732 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:00.717109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.3:47607)
0924 11:48:00.717954 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:00.717997 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:00.720060 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:00.720911 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:00.720920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:00.721468 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:00.722035 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:00.722052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.723073 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.723077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:00.724250 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:00.724261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:00.724903 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:00.724907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:00.725176 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:00.725746 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:00.725767 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:00.727791 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:00.730727 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:00.730733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:00.730746 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:00.731264 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:00.731676 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:00.731681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:00.731685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:00.731961 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:00.732568 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:00.732576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:00.732981 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 11:48:00.733282 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 11:48:00.733643 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":363,"threads_started":1}
I20260924 11:48:00.736249 8466 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43409
I20260924 11:48:00.736555 8466 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:00.737361 8466 heartbeater.cc:511] Master 127.1.210.62:43409 requested a full tablet report, sending...
I20260924 11:48:00.738692 8010 ts_manager.cc:194] Registered new tserver with Master: cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:00.739462 8010 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:47607
I20260924 11:48:00.745012 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:48:00.763024 8199 heartbeater.cc:503] Master 127.1.210.62:43409 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.1.210.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.61.keytab.
Entry for principal oryx/127.1.210.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.61.keytab.
I20260924 11:48:00.792068 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:37079
--webserver_interface=127.1.210.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.61.keytab
--principal=oryx/127.1.210.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43409,127.1.210.61:37079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:01.139989 8471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:01.140401 8471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:01.140551 8471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:01.149950 8471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:01.150060 8471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:01.150112 8471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:01.150175 8471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:01.150228 8471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:01.161417 8471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43409,127.1.210.61:37079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.61.keytab
--principal=oryx/127.1.210.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:37079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
W20260924 11:48:01.161785 8471 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.210.62:43409,127.1.210.61:37079'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 11:48:01.162827 8471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:01.164141 8471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:01.177645 8477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:01.177675 8476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:01.178439 8479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:01.179201 8471 server_base.cc:1034] running on GCE node
I20260924 11:48:01.179937 8471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:01.181329 8471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:01.182559 8471 hybrid_clock.cc:648] HybridClock initialized: now 1790250481182491 us; error 54 us; skew 500 ppm
I20260924 11:48:01.182888 8471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:01.186372 8471 init.cc:377] Logged in from keytab as oryx/127.1.210.61@KRBTEST.COM (short username oryx)
I20260924 11:48:01.187841 8471 webserver.cc:460] Webserver started at http://127.1.210.61:46109/ using document root <none> and password file <none>
I20260924 11:48:01.188527 8471 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:01.188655 8471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:01.188953 8471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:01.191859 8471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/instance:
uuid: "ebb5374cc6164b31acbed1081a4843e7"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "c573280e7ec42b2ec8d18c78f7b0344d"
server_key_iv: "02b5f733b475814b2e9d57d62895e7f7"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.192646 8471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal/instance:
uuid: "ebb5374cc6164b31acbed1081a4843e7"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "c573280e7ec42b2ec8d18c78f7b0344d"
server_key_iv: "02b5f733b475814b2e9d57d62895e7f7"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.197806 8471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:48:01.201284 8486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:01.202538 8471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:48:01.202714 8471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
uuid: "ebb5374cc6164b31acbed1081a4843e7"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "c573280e7ec42b2ec8d18c78f7b0344d"
server_key_iv: "02b5f733b475814b2e9d57d62895e7f7"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.202922 8471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:01.214887 8471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:01.218811 8471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:01.219071 8471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:01.237021 8471 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.61:37079
I20260924 11:48:01.237046 8537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.61:37079 every 8 connection(s)
I20260924 11:48:01.238415 8471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
I20260924 11:48:01.242446 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:01.242511 8333 heartbeater.cc:503] Master 127.1.210.62:43409 was elected leader, sending a full tablet report...
I20260924 11:48:01.245550 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8471
I20260924 11:48:01.250811 8538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.61@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:01.268215 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.248097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.1.210.61:37079)
0924 11:48:01.248963 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.248975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.249013 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:48:01.250668 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.250689 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.251099 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.251418 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.253168 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.255222 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.256609 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.257340 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.257567 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.261540 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.261547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.261558 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.261586 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.264307 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.264959 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.264974 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.264980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.265190 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.265578 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.265585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.265588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.266095 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.266401 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.266847 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.267229 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":393,"threads_started":1}
I20260924 11:48:01.268338 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.253689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.1.210.62:43409)
0924 11:48:01.254867 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.254873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.254891 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:48:01.254944 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.254949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.255019 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.255134 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.256352 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.257186 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.258721 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.259138 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.259266 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.262762 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.262767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.262771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.262801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.265567 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.266299 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.266304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.266314 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.266434 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.266801 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.266808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.266812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.267160 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.267328 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.267753 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.267962 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":974,"threads_started":1}
I20260924 11:48:01.268437 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.253650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:38544)
0924 11:48:01.254456 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.254510 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.254692 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.255232 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.255236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.255944 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.256181 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.256189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.257318 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.257323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.258575 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.258587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.259369 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.259376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.259520 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.260271 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.260297 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.262567 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.265740 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.265751 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.265760 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.266175 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.266540 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.266547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.266553 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.266690 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.267295 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.267302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.267465 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.267657 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.267954 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":275,"threads_started":1}
I20260924 11:48:01.271070 8538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.61@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
I20260924 11:48:01.282363 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.271743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40364)
0924 11:48:01.271956 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.271975 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.272073 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.272351 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.272354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.272653 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.272775 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.272783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.273867 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.273870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.274722 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.274732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.275342 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.275346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.275470 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.276053 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.276069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.277951 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.280415 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.280418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.280421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.280763 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.281058 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.281121 (+ 63us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.281124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.281239 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.281707 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.281710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.281833 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.281937 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.282127 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:48:01.282562 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.271975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.1.210.61:37079)
0924 11:48:01.272111 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.272115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.272129 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:48:01.272164 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.272167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.272215 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.272343 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.272885 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.273748 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.274837 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.275227 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.275321 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.278096 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.278100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.278103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.278127 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.280299 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.280857 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.280860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.280862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.280959 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.281330 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.281333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.281334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.281612 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.281716 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.282012 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.282238 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":57}
I20260924 11:48:01.288928 8538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Bootstrap starting.
I20260924 11:48:01.292598 8538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:01.293663 8538 log.cc:824] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:01.296141 8538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: No bootstrap required, opened a new log
I20260924 11:48:01.303023 8538 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } }
I20260924 11:48:01.303335 8538 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:01.303431 8538 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb5374cc6164b31acbed1081a4843e7, State: Initialized, Role: FOLLOWER
I20260924 11:48:01.304018 8538 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } }
I20260924 11:48:01.304782 8545 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } } }
I20260924 11:48:01.305097 8545 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:48:01.305755 8538 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:48:01.314136 8556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:48:01.314333 8556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:48:01.316704 8471 master_runner.cc:428] Detected that this master ebb5374cc6164b31acbed1081a4843e7 is joining an existing cluster
I20260924 11:48:01.316810 8471 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.210.61:37079
I20260924 11:48:01.338958 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.320862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.1.210.62:43409)
0924 11:48:01.321088 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.321094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.321676 (+ 582us) server_negotiation.cc:407] Connection header received
0924 11:48:01.321940 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.321946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.322035 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.322138 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.323544 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.324576 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.327875 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.328546 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.328644 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.331285 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.331292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.331297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.331334 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.334880 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.336054 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.336061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.336064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.336222 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.336705 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.336710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.336714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.337387 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.337523 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.338324 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.338563 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:48:01.340279 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.320816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:38550)
0924 11:48:01.321572 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.321591 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.321779 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.322310 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.322315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.322991 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.323369 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.323386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.324751 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.324757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.327699 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.327712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.328932 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.328939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.329220 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.329901 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.329924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.331070 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.334998 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.335001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.335004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.335635 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.336380 (+ 745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.336385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.336391 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.336568 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.337624 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.337630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.337887 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.339320 (+ 1433us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.339717 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":41,"thread_start_us":423,"threads_started":1}
I20260924 11:48:01.342420 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.322277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.1.210.61:37079)
0924 11:48:01.322761 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.322770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.324190 (+ 1420us) server_negotiation.cc:407] Connection header received
0924 11:48:01.324382 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.324387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.324446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.324524 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.326206 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.327324 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.329443 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.330627 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.330734 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.333704 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.333717 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.333722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.333746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.338840 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.339948 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.339953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.339957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.340094 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.340869 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.340873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.340875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.341222 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.341394 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.341885 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.342112 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":299}
I20260924 11:48:01.344511 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.321744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40380)
0924 11:48:01.324107 (+ 2363us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.324124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.324269 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.324711 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.324716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.325672 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.326044 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.326057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.327527 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.327534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.329300 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.329316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.331036 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.331043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.331250 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.332153 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.332178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.333553 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.338975 (+ 5422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.338982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.338986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.339816 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.340221 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.340227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.340230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.340758 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.341456 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.341459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.341595 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.342622 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.342891 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2254,"mutex_wait_us":4,"thread_start_us":2073,"threads_started":1}
WARNING: no policy specified for oryx/127.1.210.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.60@KRBTEST.COM" created.
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250479, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.62@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
I20260924 11:48:01.363739 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.347995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.1.210.61:37079)
0924 11:48:01.348303 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.348310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.348738 (+ 428us) server_negotiation.cc:407] Connection header received
0924 11:48:01.348981 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.348984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.349121 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.349236 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.350594 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.351798 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.353719 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.354556 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.354695 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.358192 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.358199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.358208 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.358234 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.361188 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.361864 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.361871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.361875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.362000 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.362363 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.362370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.362373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.362609 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.362775 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.363193 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.363392 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:48:01.363792 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.347972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40386)
0924 11:48:01.348594 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.348634 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.348805 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.349301 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.349308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.350139 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.350412 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.350424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.351942 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.351948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.353561 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.353581 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.354729 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.354735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.354928 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.355631 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.355670 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.358005 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.361310 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.361321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.361329 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.361745 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.362115 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.362121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.362125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.362259 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.362754 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.362762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.362925 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.363116 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.363400 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":222,"threads_started":1}
I20260924 11:48:01.365824 8010 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.210.61:37079: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7"
member_type: NON_VOTER
last_known_addr {
host: "127.1.210.61"
port: 37079
}
attrs {
promote: true
}
}
dest_uuid: "5aa51beb1207440099d8e59611229b44"
cas_config_opid_index: -1
I20260924 11:48:01.366729 8010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.60.keytab.
Entry for principal oryx/127.1.210.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.60.keytab.
I20260924 11:48:01.378384 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:43477
--webserver_interface=127.1.210.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.60.keytab
--principal=oryx/127.1.210.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:48:01.383155 8513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:01.382882 8569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ebb5374cc6164b31acbed1081a4843e7 (127.1.210.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } }
I20260924 11:48:01.386021 8569 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.210.61:37079
I20260924 11:48:01.386740 8471 master.cc:537] Master@127.1.210.61:37079 shutting down...
I20260924 11:48:01.387940 8568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } }
I20260924 11:48:01.388247 8568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:01.386914 8513 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Refusing update from remote peer 5aa51beb1207440099d8e59611229b44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 11:48:01.390929 8568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:48:01.394258 7994 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 11:48:01.401356 8471 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:01.401973 8471 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:01.402293 8471 tablet_replica.cc:330] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: stopping tablet replica
I20260924 11:48:01.410467 8471 master.cc:552] Master@127.1.210.61:37079 shutdown complete.
I20260924 11:48:01.410607 8471 master_runner.cc:305] Clearing existing system tablet
I20260924 11:48:01.411144 8471 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:01.411310 8471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:01.415964 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.415499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40398)
0924 11:48:01.415676 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.415781 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:37079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 11:48:01.416282 7994 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 -> Peer ebb5374cc6164b31acbed1081a4843e7 (127.1.210.61:37079): Couldn't send request to peer ebb5374cc6164b31acbed1081a4843e7. Status: Network error: Client connection negotiation failed: client connection to 127.1.210.61:37079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 11:48:01.418176 8471 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 11:48:01.419718 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:01.420094 8471 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:01.420333 8471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 11:48:01.420413 8471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
uuid: "ebb5374cc6164b31acbed1081a4843e7"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "c573280e7ec42b2ec8d18c78f7b0344d"
server_key_iv: "02b5f733b475814b2e9d57d62895e7f7"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.425330 8471 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:01.431663 8471 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 11:48:01.431834 8471 log.cc:1196] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 11:48:01.432241 8471 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Deleting consensus metadata
I20260924 11:48:01.432518 8471 master_runner.cc:315] Copying system tablet from 127.1.210.62:43409
I20260924 11:48:01.435505 8471 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: tablet copy: Beginning tablet copy session from remote peer at address 127.1.210.62:43409
I20260924 11:48:01.451398 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.436215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:38556)
0924 11:48:01.436907 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.436932 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.437124 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.437519 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.437524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.438017 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.438315 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.438328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.439724 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.439732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.441335 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.441349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.442385 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.442391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.442558 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.443287 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.443309 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.444368 (+ 1059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.448472 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.448477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.448481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.448969 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.449449 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.449454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.449458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.449619 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.450309 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.450313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.450484 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.450625 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.450933 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":442,"threads_started":1}
I20260924 11:48:01.451570 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.436357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.1.210.62:43409)
0924 11:48:01.436531 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.436537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.437046 (+ 509us) server_negotiation.cc:407] Connection header received
0924 11:48:01.437273 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.437279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.437362 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.437474 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.438468 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.439566 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.441509 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.442202 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.442328 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.444625 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.444633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.444638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.444675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.448312 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.449156 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.449163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.449167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.449301 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.449745 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.449751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.449755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.450103 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.450233 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.450827 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.451108 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:48:01.452812 8030 tablet_copy_service.cc:140] P 5aa51beb1207440099d8e59611229b44: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ebb5374cc6164b31acbed1081a4843e7 ({username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:38556)
I20260924 11:48:01.453161 8030 tablet_copy_service.cc:161] P 5aa51beb1207440099d8e59611229b44: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ebb5374cc6164b31acbed1081a4843e7 at {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:38556: session id = ebb5374cc6164b31acbed1081a4843e7-00000000000000000000000000000000
I20260924 11:48:01.457537 8030 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 11:48:01.459746 8471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 11:48:01.466681 8471 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: tablet copy: Starting download of 0 data blocks...
I20260924 11:48:01.466992 8471 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: tablet copy: Starting download of 1 WAL segments...
I20260924 11:48:01.469700 8471 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 11:48:01.474344 8030 tablet_copy_service.cc:342] P 5aa51beb1207440099d8e59611229b44: Request end of tablet copy session ebb5374cc6164b31acbed1081a4843e7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:38556
I20260924 11:48:01.474633 8030 tablet_copy_service.cc:433] P 5aa51beb1207440099d8e59611229b44: ending tablet copy session ebb5374cc6164b31acbed1081a4843e7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ebb5374cc6164b31acbed1081a4843e7
I20260924 11:48:01.486301 8471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:01.489146 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:01.494760 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:01.499010 8471 server_base.cc:1034] running on GCE node
W20260924 11:48:01.499437 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:01.499869 8471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:01.500524 8471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:01.501744 8471 hybrid_clock.cc:648] HybridClock initialized: now 1790250481501686 us; error 33 us; skew 500 ppm
I20260924 11:48:01.501958 8471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:01.504154 8471 init.cc:377] Logged in from keytab as oryx/127.1.210.61@KRBTEST.COM (short username oryx)
I20260924 11:48:01.505385 8471 webserver.cc:460] Webserver started at http://127.1.210.61:46629/ using document root <none> and password file <none>
I20260924 11:48:01.505752 8471 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:01.505832 8471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:01.508787 8471 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 11:48:01.511241 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:01.511703 8471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260924 11:48:01.511843 8471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
uuid: "ebb5374cc6164b31acbed1081a4843e7"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "c573280e7ec42b2ec8d18c78f7b0344d"
server_key_iv: "02b5f733b475814b2e9d57d62895e7f7"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.512017 8471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:01.544852 8471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:01.549144 8471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:01.589401 8471 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.61:37079
I20260924 11:48:01.589424 8642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.61:37079 every 8 connection(s)
I20260924 11:48:01.590401 8471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
I20260924 11:48:01.597836 8643 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:48:01.601534 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Bootstrap starting.
I20260924 11:48:01.620051 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:48:01.620548 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Bootstrap complete.
I20260924 11:48:01.621582 8643 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } }
I20260924 11:48:01.621869 8643 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebb5374cc6164b31acbed1081a4843e7, State: Initialized, Role: LEARNER
I20260924 11:48:01.622130 8643 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } }
I20260924 11:48:01.622936 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } }
I20260924 11:48:01.623242 8643 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:01.623348 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:48:01.637842 8657 catalog_manager.cc:1259] Loaded cluster ID: 45533faa9e354fc9aa415961c72bdb5e
I20260924 11:48:01.637951 8657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: loading cluster ID for follower catalog manager: success
I20260924 11:48:01.642309 8657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: acquiring CA information for follower catalog manager: success
I20260924 11:48:01.645126 8657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:48:01.741791 8466 heartbeater.cc:503] Master 127.1.210.62:43409 was elected leader, sending a full tablet report...
W20260924 11:48:01.800343 8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:01.800616 8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:01.800702 8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:01.808027 8570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:01.808130 8570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:01.808189 8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:01.808245 8570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:01.808288 8570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:01.818508 8570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.60.keytab
--principal=oryx/127.1.210.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:43477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:01.819825 8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:01.821034 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:01.832988 8662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:01.833114 8665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:01.833176 8663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:01.834524 8570 server_base.cc:1034] running on GCE node
I20260924 11:48:01.835216 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:01.836606 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:01.837808 8570 hybrid_clock.cc:648] HybridClock initialized: now 1790250481837771 us; error 26 us; skew 500 ppm
I20260924 11:48:01.838150 8570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:01.855350 8570 init.cc:377] Logged in from keytab as oryx/127.1.210.60@KRBTEST.COM (short username oryx)
I20260924 11:48:01.856911 8570 webserver.cc:460] Webserver started at http://127.1.210.60:37991/ using document root <none> and password file <none>
I20260924 11:48:01.857676 8570 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:01.857810 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:01.858094 8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:01.860962 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/instance:
uuid: "34e8b054e27145cda2ad94ba4c0f1527"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "f905a41d4e0fc07003aa9f0e2baf368f"
server_key_iv: "070d7c3ce845857faede843bb75bff6f"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.861861 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal/instance:
uuid: "34e8b054e27145cda2ad94ba4c0f1527"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "f905a41d4e0fc07003aa9f0e2baf368f"
server_key_iv: "070d7c3ce845857faede843bb75bff6f"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.866681 8570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:48:01.870217 8672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:01.871577 8570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:48:01.871733 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
uuid: "34e8b054e27145cda2ad94ba4c0f1527"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "f905a41d4e0fc07003aa9f0e2baf368f"
server_key_iv: "070d7c3ce845857faede843bb75bff6f"
server_key_version: "encryptionkey@0"
I20260924 11:48:01.871946 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:01.884146 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:01.888355 8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:01.888635 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:01.907246 8570 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.60:43477
I20260924 11:48:01.907266 8723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.60:43477 every 8 connection(s)
I20260924 11:48:01.908581 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
I20260924 11:48:01.910322 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8570
I20260924 11:48:01.912506 8724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:01.919624 8724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.60@KRBTEST.COM
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.60@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:01.933785 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.912319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.1.210.60:43477)
0924 11:48:01.913230 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.913242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.913282 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:01.914846 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.914865 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.915285 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.915673 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.917632 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.918881 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.920131 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.920831 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.921016 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.925990 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.925998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.926007 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.926038 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.929729 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.930533 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.930539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.930545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.930694 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.931119 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.931122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.931124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.931569 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.931860 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.932294 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.932727 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":425,"threads_started":1}
I20260924 11:48:01.936599 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.922077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.1.210.62:43409)
0924 11:48:01.922241 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.922245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.922802 (+ 557us) server_negotiation.cc:407] Connection header received
0924 11:48:01.922992 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.922999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.923094 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.923187 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.924269 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.925311 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.926671 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.927333 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.927463 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.931259 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.931269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.931274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.931302 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.933800 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.934643 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.934647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.934649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.934782 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.935181 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.935186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.935190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.935431 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.935593 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.936024 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.936238 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:48:01.936642 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.922027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:38558)
0924 11:48:01.922672 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.922705 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.922870 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.923240 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.923248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.923884 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.924114 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.924122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.925442 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.925449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.926526 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.926542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.927503 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.927508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.927704 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.928773 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.928792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.931083 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.933933 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.933943 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.933950 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.934527 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.934894 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.934898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.934901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.935071 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.935556 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.935563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.935735 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.935928 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.936235 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":335,"threads_started":1}
I20260924 11:48:01.937836 8724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.60@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
I20260924 11:48:01.950644 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.938378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40404)
0924 11:48:01.938565 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.938580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.938650 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.939338 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.939342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.939699 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.939817 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.939825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.941083 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.941091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.941995 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.942005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.942677 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.942682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.942868 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.943349 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.943365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.945231 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.948369 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.948376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.948380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.948739 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.949144 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.949149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.949153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.949280 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.949759 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.949765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.949906 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.950038 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.950312 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:48:01.950665 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.938569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.1.210.61:37079)
0924 11:48:01.939085 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.939090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.939104 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:48:01.939150 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.939153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.939206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.939319 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.939995 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.940929 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.942131 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.942532 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.942667 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.945412 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.945417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.945420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.945449 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.948248 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.948852 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.948858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.948862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.948988 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.949382 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.949388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.949392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.949646 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.949804 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.950130 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.950320 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260924 11:48:01.951798 8724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:48:01 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250481, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.60@KRBTEST.COM for oryx/127.1.210.60@KRBTEST.COM
I20260924 11:48:01.962981 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.952378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.0.0.1:54746)
0924 11:48:01.952575 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:01.952593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:01.952681 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:01.952916 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:01.952919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:01.953270 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:01.953399 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.953406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.954272 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.954275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.955053 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:01.955062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.955672 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.955676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:01.955798 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.956282 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:01.956304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:01.958183 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:01.960869 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.960877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.960882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.961265 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.961741 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:01.961744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:01.961746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:01.961897 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:01.962322 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:01.962325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:01.962445 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 11:48:01.962545 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:48:01.962709 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 11:48:01.963140 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:01.952579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.1.210.60:43477)
0924 11:48:01.952709 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:01.952713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:01.952726 (+ 13us) server_negotiation.cc:407] Connection header received
0924 11:48:01.952762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:01.952765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:01.952811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:01.952885 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:01.953509 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.954168 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.955161 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:01.955542 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:01.955634 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:01.958341 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:01.958346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:01.958349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:01.958375 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:01.960745 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.961415 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.961422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.961426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.961624 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:01.961991 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:01.961994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:01.961996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:01.962230 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:01.962330 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:01.962634 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 11:48:01.962824 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47}
I20260924 11:48:01.967574 8724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Bootstrap starting.
I20260924 11:48:01.971154 8724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:01.972093 8724 log.cc:824] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:01.974627 8724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: No bootstrap required, opened a new log
I20260924 11:48:01.981638 8724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } }
I20260924 11:48:01.981951 8724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:01.982059 8724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e8b054e27145cda2ad94ba4c0f1527, State: Initialized, Role: FOLLOWER
I20260924 11:48:01.982656 8724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } }
I20260924 11:48:01.983467 8731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } } }
I20260924 11:48:01.983752 8731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:48:01.984447 8724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:48:01.993532 8742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:48:01.994055 8742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:48:01.997601 8570 master_runner.cc:428] Detected that this master 34e8b054e27145cda2ad94ba4c0f1527 is joining an existing cluster
I20260924 11:48:01.997715 8570 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.210.60:43477
I20260924 11:48:02.021229 8731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:02.021571 8731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } }
I20260924 11:48:02.023407 8731 leader_election.cc:288] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409), ebb5374cc6164b31acbed1081a4843e7 (127.1.210.61:37079)
I20260924 11:48:02.024663 8020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34e8b054e27145cda2ad94ba4c0f1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa51beb1207440099d8e59611229b44" is_pre_election: true
I20260924 11:48:02.024773 8618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34e8b054e27145cda2ad94ba4c0f1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb5374cc6164b31acbed1081a4843e7" is_pre_election: true
I20260924 11:48:02.025488 8618 raft_consensus.cc:1799] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Handling vote request from an unknown peer 34e8b054e27145cda2ad94ba4c0f1527
I20260924 11:48:02.025756 8618 raft_consensus.cc:2396] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 34e8b054e27145cda2ad94ba4c0f1527 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 11:48:02.026305 8673 leader_election.cc:302] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e8b054e27145cda2ad94ba4c0f1527; no voters: 5aa51beb1207440099d8e59611229b44, ebb5374cc6164b31acbed1081a4843e7
I20260924 11:48:02.026959 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.001747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.1.210.62:43409)
0924 11:48:02.001958 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.001966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.002562 (+ 596us) server_negotiation.cc:407] Connection header received
0924 11:48:02.002822 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.002828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.002905 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.002975 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.006349 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.007559 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.009875 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.010517 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.010613 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.013711 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.013718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.013722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.013760 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.017945 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.018980 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.018989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.018995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.019189 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.019687 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.019695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.019701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.020150 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.020302 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.026202 (+ 5900us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.026540 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:48:02.027047 8731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:02.028481 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.001832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:38574)
0924 11:48:02.002458 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.002479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.002668 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.003207 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.003214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.005777 (+ 2563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.006182 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.006193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.007753 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.007758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.009719 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.009730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.010876 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.010884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.011092 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.012170 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.012193 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.013497 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.018165 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.018173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.018180 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.018831 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.019347 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.019355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.019361 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.019557 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.020393 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.020400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.020655 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.027378 (+ 6723us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.027795 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":140,"thread_start_us":334,"threads_started":1}
I20260924 11:48:02.031767 8731 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 11:48:02.036700 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.002779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.1.210.61:37079)
0924 11:48:02.002997 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.003005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.003607 (+ 602us) server_negotiation.cc:407] Connection header received
0924 11:48:02.004375 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.004382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.004478 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.004592 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.012198 (+ 7606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.013732 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.015753 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.016395 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.016489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.018900 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.018906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.018911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.018948 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.022450 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.023236 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.023242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.023246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.023384 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.023867 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.023873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.023876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.024228 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.024349 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.036114 (+ 11765us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.036360 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:48:02.036765 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.002567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40418)
0924 11:48:02.003509 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.003525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.004122 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.004798 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.004803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.005579 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.011964 (+ 6385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.011979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.013965 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.013970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.015584 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.015598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.016762 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.016768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.016946 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.017528 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.017563 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.018708 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.022634 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.022639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.022643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.023096 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.023535 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.023551 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.023555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.023728 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.029351 (+ 5623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.029354 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.031172 (+ 1818us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.035964 (+ 4792us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.036314 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":635,"threads_started":1}
I20260924 11:48:02.038272 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.004592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.1.210.60:43477)
0924 11:48:02.004768 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.004775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.004843 (+ 68us) server_negotiation.cc:407] Connection header received
0924 11:48:02.005035 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.005040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.005123 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.005224 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.006773 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.007772 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.009866 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.010521 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.010613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.030965 (+ 20352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.030977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.030981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.031015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.034994 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.035782 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.035790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.035796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.035947 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.036334 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.036341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.036349 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.036660 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.037233 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.037621 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.037929 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":69}
I20260924 11:48:02.038883 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.003523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.0.0.1:54762)
0924 11:48:02.004768 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.004784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.004897 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.005575 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.005579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.006289 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.006630 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.006641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.007958 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.007968 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.009717 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.009730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.011092 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.011097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.011304 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.025500 (+ 14196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.025539 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.029823 (+ 4284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.035142 (+ 5319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.035147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.035151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.035636 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.036069 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.036076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.036080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.036213 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.037208 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.037212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.037350 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.037517 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.038411 (+ 894us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":60,"thread_start_us":1639,"threads_started":1}
I20260924 11:48:02.039656 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8071
I20260924 11:48:02.049268 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.013685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.1.210.61:37079)
0924 11:48:02.036993 (+ 23308us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.036998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.041841 (+ 4843us) server_negotiation.cc:407] Connection header received
0924 11:48:02.042086 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.042091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.042177 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.042258 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:02.042919 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.043983 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.046510 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.047855 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.047963 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.048190 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.048830 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.048959 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23196,"thread_start_us":27547,"threads_started":1}
I20260924 11:48:02.050330 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.013231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:40426)
0924 11:48:02.041748 (+ 28517us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.041768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.041940 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.042463 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.042468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.042493 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:02.042762 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.042774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.044166 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.044171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.046331 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.046345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.048400 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.048406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.048614 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.048673 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.049444 (+ 771us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.049597 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28395,"thread_start_us":28540,"threads_started":1}
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250479, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.62@KRBTEST.COM for oryx/127.1.210.60@KRBTEST.COM
I20260924 11:48:02.067027 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.042169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.1.210.60:43477)
0924 11:48:02.043570 (+ 1401us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.043577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.050666 (+ 7089us) server_negotiation.cc:407] Connection header received
0924 11:48:02.050939 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.050945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.051051 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.051121 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.052233 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.053396 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.055157 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.055841 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.055956 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.060391 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.060397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.060402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.060437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.064023 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.064862 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.064868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.064872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.065010 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.065448 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.065454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.065457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.065809 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.065926 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.066447 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.066680 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298}
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.1@KRBTEST.COM
I20260924 11:48:02.067870 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.043714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.0.0.1:54768)
0924 11:48:02.050581 (+ 6867us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.050600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.050769 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.051353 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.051358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.051873 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.052062 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.052075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.053577 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.053583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.054998 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.055012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.056171 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.056176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.056359 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.057026 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.057051 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.060190 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.064194 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.064201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.064204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.064727 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.065172 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.065177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.065181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.065317 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.066016 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.066022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.066187 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.067217 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.067485 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6759,"threads_started":1}
I20260924 11:48:02.070791 8618 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 11:48:02.071455 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa51beb1207440099d8e59611229b44. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } }
I20260924 11:48:02.071723 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:48:02.072171 8010 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.210.60:43477: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527"
member_type: NON_VOTER
last_known_addr {
host: "127.1.210.60"
port: 43477
}
attrs {
promote: true
}
}
dest_uuid: "5aa51beb1207440099d8e59611229b44"
cas_config_opid_index: 5
I20260924 11:48:02.073163 8010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } }
I20260924 11:48:02.075203 8751 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.078461 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.041463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.1.210.1:44009)
0924 11:48:02.052354 (+ 10891us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.052367 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.052407 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:02.052499 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.052513 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.052858 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.055459 (+ 2601us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.055757 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.057803 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.060470 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.061686 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.068074 (+ 6388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.069108 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.069123 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.069143 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.069183 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.074321 (+ 5138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.075263 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.075271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.075275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.075412 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.075834 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.075839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.075841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.076318 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.077652 (+ 1334us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.077805 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.078053 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10716,"thread_start_us":11600,"threads_started":1}
I20260924 11:48:02.079579 8751 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.210.60:43477
I20260924 11:48:02.079916 8194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:02.080549 8570 master.cc:537] Master@127.1.210.60:43477 shutting down...
I20260924 11:48:02.087096 8618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Refusing update from remote peer 5aa51beb1207440099d8e59611229b44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260924 11:48:02.093564 7994 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 -> Peer 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60:43477): Couldn't send request to peer 34e8b054e27145cda2ad94ba4c0f1527. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 11:48:02.094521 8756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 11:48:02.107230 8757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.107681 8757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:02.108274 8618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.114687 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: NON_VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: true } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.114950 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:48:02.115576 8751 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: attempting to promote NON_VOTER ebb5374cc6164b31acbed1081a4843e7 to VOTER
I20260924 11:48:02.115898 8570 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:02.116276 8570 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:02.116487 8570 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: stopping tablet replica
I20260924 11:48:02.116505 8751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } }
I20260924 11:48:02.119402 8618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 LEARNER]: Refusing update from remote peer 5aa51beb1207440099d8e59611229b44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 11:48:02.120131 8756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 11:48:02.120518 7994 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 -> Peer 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60:43477): Couldn't send request to peer 34e8b054e27145cda2ad94ba4c0f1527. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 11:48:02.125231 8570 master.cc:552] Master@127.1.210.60:43477 shutdown complete.
I20260924 11:48:02.125320 8570 master_runner.cc:305] Clearing existing system tablet
I20260924 11:48:02.125639 8570 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:02.125746 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:02.128038 8756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ebb5374cc6164b31acbed1081a4843e7 (127.1.210.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.132288 8618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ebb5374cc6164b31acbed1081a4843e7 (127.1.210.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.135946 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.136186 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:48:02.136507 8757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.136446 8756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.136726 8757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:02.136749 8756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:02.137390 8570 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.002s sys 0.001s
I20260924 11:48:02.141916 8764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:02.145409 8570 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:02.145689 8570 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260924 11:48:02.145792 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
uuid: "34e8b054e27145cda2ad94ba4c0f1527"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "f905a41d4e0fc07003aa9f0e2baf368f"
server_key_iv: "070d7c3ce845857faede843bb75bff6f"
server_key_version: "encryptionkey@0"
I20260924 11:48:02.150975 8570 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:02.158223 8570 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 11:48:02.158384 8570 log.cc:1196] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 11:48:02.158896 8570 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Deleting consensus metadata
I20260924 11:48:02.159152 8570 master_runner.cc:315] Copying system tablet from 127.1.210.62:43409
I20260924 11:48:02.166908 8570 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: tablet copy: Beginning tablet copy session from remote peer at address 127.1.210.62:43409
I20260924 11:48:02.182771 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.167705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.1.210.62:43409)
0924 11:48:02.167987 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.167995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.168491 (+ 496us) server_negotiation.cc:407] Connection header received
0924 11:48:02.168668 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.168674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.168752 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.168852 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.169916 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.171096 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.173108 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.173760 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.173890 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.176573 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.176580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.176585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.176623 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.179897 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.180644 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.180650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.180653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.180779 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.181227 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.181232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.181235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.181561 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.181700 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.182222 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.182442 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 11:48:02.184098 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.167660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:38582)
0924 11:48:02.168368 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.168392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.168533 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.168900 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.168904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.169476 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.169755 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.169767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.171254 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.171259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.172895 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.172909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.174133 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.174139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.174343 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.175073 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.175094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.176375 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.180039 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.180044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.180048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.180512 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.180904 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.180908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.180912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.181111 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.181796 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.181804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.181959 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.183305 (+ 1346us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.183630 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":457,"threads_started":1}
I20260924 11:48:02.184918 8030 tablet_copy_service.cc:140] P 5aa51beb1207440099d8e59611229b44: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 34e8b054e27145cda2ad94ba4c0f1527 ({username='oryx', principal='oryx/127.1.210.60@KRBTEST.COM'} at 127.0.0.1:38582)
I20260924 11:48:02.185106 8030 tablet_copy_service.cc:161] P 5aa51beb1207440099d8e59611229b44: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 34e8b054e27145cda2ad94ba4c0f1527 at {username='oryx', principal='oryx/127.1.210.60@KRBTEST.COM'} at 127.0.0.1:38582: session id = 34e8b054e27145cda2ad94ba4c0f1527-00000000000000000000000000000000
I20260924 11:48:02.188727 8030 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 11:48:02.190588 8570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 11:48:02.197716 8570 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: tablet copy: Starting download of 0 data blocks...
I20260924 11:48:02.197985 8570 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: tablet copy: Starting download of 1 WAL segments...
I20260924 11:48:02.200439 8570 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 11:48:02.204593 8030 tablet_copy_service.cc:342] P 5aa51beb1207440099d8e59611229b44: Request end of tablet copy session 34e8b054e27145cda2ad94ba4c0f1527-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.210.60@KRBTEST.COM'} at 127.0.0.1:38582
I20260924 11:48:02.204728 8030 tablet_copy_service.cc:433] P 5aa51beb1207440099d8e59611229b44: ending tablet copy session 34e8b054e27145cda2ad94ba4c0f1527-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 34e8b054e27145cda2ad94ba4c0f1527
I20260924 11:48:02.217356 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:02.220317 8771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:02.220594 8772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:02.228472 8774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:02.233397 8570 server_base.cc:1034] running on GCE node
I20260924 11:48:02.233738 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:02.234336 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:02.235533 8570 hybrid_clock.cc:648] HybridClock initialized: now 1790250482235480 us; error 44 us; skew 500 ppm
I20260924 11:48:02.235787 8570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:02.238003 8570 init.cc:377] Logged in from keytab as oryx/127.1.210.60@KRBTEST.COM (short username oryx)
I20260924 11:48:02.241580 8570 webserver.cc:460] Webserver started at http://127.1.210.60:35145/ using document root <none> and password file <none>
I20260924 11:48:02.241922 8570 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:02.242020 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:02.246716 8570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:02.249025 8781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:02.249531 8570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 11:48:02.249702 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
uuid: "34e8b054e27145cda2ad94ba4c0f1527"
format_stamp: "Formatted at 2026-09-24 11:48:01 on dist-test-slave-dd32"
server_key: "f905a41d4e0fc07003aa9f0e2baf368f"
server_key_iv: "070d7c3ce845857faede843bb75bff6f"
server_key_version: "encryptionkey@0"
I20260924 11:48:02.249886 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:02.251804 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8071
I20260924 11:48:02.268045 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:02.270254 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:44009
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:43409
--webserver_port=35837
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:48:02.272148 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:02.306031 8570 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.60:43477
I20260924 11:48:02.306079 8833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.60:43477 every 8 connection(s)
I20260924 11:48:02.306896 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
I20260924 11:48:02.313848 8834 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:48:02.318070 8834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Bootstrap starting.
I20260924 11:48:02.331728 8834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:48:02.332198 8834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: Bootstrap complete.
I20260924 11:48:02.333122 8834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } }
I20260924 11:48:02.333364 8834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34e8b054e27145cda2ad94ba4c0f1527, State: Initialized, Role: LEARNER
I20260924 11:48:02.333590 8834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } }
I20260924 11:48:02.334357 8837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.334651 8834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:02.334766 8837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:48:02.343137 8848 catalog_manager.cc:1259] Loaded cluster ID: 45533faa9e354fc9aa415961c72bdb5e
I20260924 11:48:02.343240 8848 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: loading cluster ID for follower catalog manager: success
I20260924 11:48:02.345536 8848 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: acquiring CA information for follower catalog manager: success
I20260924 11:48:02.347481 8848 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:48:02.666463 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:02.666795 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:02.666970 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:02.674932 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:02.675078 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:02.675240 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:02.684813 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:44009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=35837
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:02.686167 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:02.687459 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:02.701113 8853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:02.701041 8856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:02.701004 8854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:02.702101 8788 server_base.cc:1034] running on GCE node
I20260924 11:48:02.702697 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:02.703636 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:02.704895 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790250482704848 us; error 46 us; skew 500 ppm
I20260924 11:48:02.705286 8788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:02.708464 8788 init.cc:377] Logged in from keytab as oryx/127.1.210.1@KRBTEST.COM (short username oryx)
I20260924 11:48:02.709993 8788 webserver.cc:460] Webserver started at http://127.1.210.1:35837/ using document root <none> and password file <none>
I20260924 11:48:02.710690 8788 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:02.710846 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:02.716367 8788 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 11:48:02.719419 8863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:02.720964 8788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:48:02.721156 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "4c3ce6361a0c481eaf81bc1b655b8a53"
format_stamp: "Formatted at 2026-09-24 11:47:59 on dist-test-slave-dd32"
server_key: "c8b605ca43fac96f9ac53b756e18ec3f"
server_key_iv: "34a3be5727cbbd4b2c263fde56a2df76"
server_key_version: "encryptionkey@0"
I20260924 11:48:02.722523 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:02.727823 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.721027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.0.0.1:52374)
0924 11:48:02.721520 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.721541 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.721691 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.722147 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.722150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.722169 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:02.722321 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.722328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.724037 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.724041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.725532 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.725545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.727035 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.727040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.727259 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.727292 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.727389 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.727526 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":194,"threads_started":1}
I20260924 11:48:02.728693 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.721194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.1.210.60:43477)
0924 11:48:02.721796 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.721803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.721825 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:48:02.721897 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.721902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.722006 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.722131 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:02.722457 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.723896 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.725689 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.726881 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.727029 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.727229 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.727631 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.727756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":287,"threads_started":1}
I20260924 11:48:02.729795 8809 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 11:48:02.730334 8837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa51beb1207440099d8e59611229b44. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:02.730610 8837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:48:02.739637 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:02.744114 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:02.744379 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:02.745618 8788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:02.746985 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:02.747097 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:02.747174 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:02.747231 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:02.781598 8788 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:44009
I20260924 11:48:02.781672 8978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:44009 every 8 connection(s)
I20260924 11:48:02.783033 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:02.785357 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8788
I20260924 11:48:02.786082 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8205
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.2@KRBTEST.COM
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.60@KRBTEST.COM
Sep 24 11:48:02 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250482, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:02.825973 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.785885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:45785 (local address 127.1.210.61:37079)
0924 11:48:02.798157 (+ 12272us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.798167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.805149 (+ 6982us) server_negotiation.cc:407] Connection header received
0924 11:48:02.805218 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.805225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.805319 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.808095 (+ 2776us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.808701 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.809791 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.812545 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.813336 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.813458 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.818143 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.818150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.818156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.818193 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.821786 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.822697 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.822703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.822708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.822849 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.823383 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.823388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.823393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.823775 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.823902 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.825356 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.825562 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12107,"thread_start_us":12282,"threads_started":1}
I20260924 11:48:02.826401 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.800704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:54839 (local address 127.1.210.60:43477)
0924 11:48:02.800942 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.800951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.807986 (+ 7035us) server_negotiation.cc:407] Connection header received
0924 11:48:02.808089 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.808095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.808166 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.808254 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.809491 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.810508 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.812525 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.813187 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.813277 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.818493 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.818499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.818504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.818544 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.822941 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.823948 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.823955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.823958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.824089 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.824859 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.824864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.824867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.825240 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.825375 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.825868 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.826083 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:48:02.826959 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.787049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.1.210.2:42945)
0924 11:48:02.802310 (+ 15261us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.802322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.802360 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:48:02.802448 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.802460 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.802760 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.803920 (+ 1160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.805544 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.807147 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.809007 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.809880 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.810831 (+ 951us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.817916 (+ 7085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.817929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.817945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.817990 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.821901 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.822774 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.822783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.822787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.822934 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.823387 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.823394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.823398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.823813 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.824098 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.826298 (+ 2200us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.826555 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15028,"thread_start_us":15851,"threads_started":1}
I20260924 11:48:02.827174 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.799242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:45589)
0924 11:48:02.799820 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.799834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.803267 (+ 3433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.805933 (+ 2666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.805941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.806868 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.807618 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.807646 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.809978 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.809984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.813023 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.813036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.814072 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.814078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.814308 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.815434 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.815455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.819338 (+ 3883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.823077 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.823081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.823083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.823893 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.824284 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.824290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.824293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.824909 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.825538 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.825543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.825683 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.825832 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.826077 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":1484,"spinlock_wait_cycles":40832,"thread_start_us":837,"threads_started":1}
I20260924 11:48:02.827493 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.786903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.1.210.1:45785)
0924 11:48:02.799536 (+ 12633us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.799607 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.803705 (+ 4098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.805805 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.805816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.806667 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.807956 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.807966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.809978 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.809984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.812364 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.812380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.813722 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.813730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.814016 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.814770 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.814795 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.817831 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.821949 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.821959 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.821983 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.822527 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.822979 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.822985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.822989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.823251 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.823977 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.823987 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.824806 (+ 819us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.825146 (+ 340us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.825671 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10584,"mutex_wait_us":110,"spinlock_wait_cycles":2304,"thread_start_us":10851,"threads_started":1}
I20260924 11:48:02.828334 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.800615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.1.210.1:54839)
0924 11:48:02.807854 (+ 7239us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:02.807873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:02.807991 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:02.808409 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:02.808413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:02.809035 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:02.809322 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.809335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.810649 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.810653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.812355 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:02.812368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.813549 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.813575 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:02.813866 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.814604 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:02.814633 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:02.818297 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:02.823084 (+ 4787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.823088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.823091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.823816 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.824229 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:02.824234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:02.824237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:02.824685 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:02.825431 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:02.825436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:02.825577 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 11:48:02.827734 (+ 2157us) client_negotiation.cc:240] Negotiation successful
0924 11:48:02.827991 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7125,"mutex_wait_us":121,"thread_start_us":6995,"threads_started":1}
I20260924 11:48:02.828382 8328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:02.828838 8988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:02.799308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:45589 (local address 127.1.210.62:43409)
0924 11:48:02.805489 (+ 6181us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:02.805498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:02.805528 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:48:02.805614 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:02.805621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:02.805728 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:02.807188 (+ 1460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:02.808154 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.809791 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.813261 (+ 3470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:02.813862 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:02.814592 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:02.819565 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:02.819571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:02.819576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:02.819612 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:02.822926 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.824012 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.824017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.824020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.824160 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:02.825087 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:02.825096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:02.825099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:02.825405 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:02.828106 (+ 2701us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:02.828228 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 11:48:02.828450 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5804,"thread_start_us":5684,"threads_started":1}
I20260924 11:48:02.830550 8981 heartbeater.cc:347] Connected to a master server at 127.1.210.61:37079
I20260924 11:48:02.830668 8980 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43409
I20260924 11:48:02.830880 8980 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:02.830878 8981 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:02.831248 8979 heartbeater.cc:347] Connected to a master server at 127.1.210.60:43477
I20260924 11:48:02.831370 8979 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:02.831930 8981 heartbeater.cc:511] Master 127.1.210.61:37079 requested a full tablet report, sending...
I20260924 11:48:02.832010 8979 heartbeater.cc:511] Master 127.1.210.60:43477 requested a full tablet report, sending...
I20260924 11:48:02.832049 8980 heartbeater.cc:511] Master 127.1.210.62:43409 requested a full tablet report, sending...
I20260924 11:48:02.834102 8010 ts_manager.cc:194] Re-registered known tserver with Master: 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009)
I20260924 11:48:02.834249 8608 ts_manager.cc:194] Registered new tserver with Master: 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009)
I20260924 11:48:02.835513 8010 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:45589
I20260924 11:48:02.835764 8799 ts_manager.cc:194] Registered new tserver with Master: 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009)
I20260924 11:48:02.914669 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8205
I20260924 11:48:02.931170 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:42945
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:43409
--webserver_port=38461
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:48:03.209508 8864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: NON_VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: true } } }
I20260924 11:48:03.209966 8864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:03.211211 8993 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: attempting to promote NON_VOTER 34e8b054e27145cda2ad94ba4c0f1527 to VOTER
I20260924 11:48:03.211757 8993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } }
I20260924 11:48:03.213989 8809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 LEARNER]: Refusing update from remote peer 5aa51beb1207440099d8e59611229b44: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 11:48:03.214099 8618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Refusing update from remote peer 5aa51beb1207440099d8e59611229b44: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 11:48:03.214601 8992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 11:48:03.215044 8864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 11:48:03.217612 8993 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.218127 8618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.221668 8864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.221930 8864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:03.222272 8993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.222640 8993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:03.223430 8998 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.223838 8998 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:48:03.224524 8864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.224768 8864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:03.225121 8809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.228209 8837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aa51beb1207440099d8e59611229b44" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:03.228509 8837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 11:48:03.257483 8991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:03.257719 8991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:03.257899 8991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:03.264904 8991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:03.265018 8991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:03.265213 8991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:03.274952 8991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:42945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=38461
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:03.276199 8991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:03.277526 8991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:03.290004 9003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:03.290046 9004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:03.290126 9006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:03.291728 8991 server_base.cc:1034] running on GCE node
I20260924 11:48:03.292289 8991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:03.293192 8991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:03.294407 8991 hybrid_clock.cc:648] HybridClock initialized: now 1790250483294364 us; error 55 us; skew 500 ppm
I20260924 11:48:03.294703 8991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:03.297736 8991 init.cc:377] Logged in from keytab as oryx/127.1.210.2@KRBTEST.COM (short username oryx)
I20260924 11:48:03.299099 8991 webserver.cc:460] Webserver started at http://127.1.210.2:38461/ using document root <none> and password file <none>
I20260924 11:48:03.299754 8991 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:03.299906 8991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:03.304814 8991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:48:03.307471 9013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:03.308861 8991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:48:03.309015 8991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "90f2d46a63fa478b82cead56cfaa6259"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "7abba75df46639e52f2de6e1b115c35c"
server_key_iv: "06544978fb83fc6c42819056ec96d7d3"
server_key_version: "encryptionkey@0"
I20260924 11:48:03.309975 8991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:03.323860 8991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:03.327715 8991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:03.328065 8991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:03.329025 8991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:03.330582 8991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:03.330668 8991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:03.330744 8991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:03.330842 8991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:03.362887 8991 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:42945
I20260924 11:48:03.362973 9125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:42945 every 8 connection(s)
I20260924 11:48:03.364316 8991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:03.372207 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 8991
I20260924 11:48:03.372485 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8338
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.60@KRBTEST.COM
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:03.394261 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.367256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:46669 (local address 127.1.210.60:43477)
0924 11:48:03.367685 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.367690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.368722 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 11:48:03.371821 (+ 3099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.371827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.371922 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:03.372049 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:03.374216 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.375755 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.379306 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.380195 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.380313 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.386816 (+ 6503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:03.386823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:03.386830 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:03.386870 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:03.390254 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.391206 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.391213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.391219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.391356 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.391836 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.391842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.391848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.392825 (+ 977us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:03.392951 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:03.393536 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 11:48:03.393749 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260924 11:48:03.395339 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.369285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:41043 (local address 127.1.210.61:37079)
0924 11:48:03.370201 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.370207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.370229 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:48:03.372337 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.372345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.372421 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:03.372536 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:03.374878 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.376371 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.379093 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.379666 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.379779 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.386068 (+ 6289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:03.386085 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:03.386091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:03.386136 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:03.389939 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.390982 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.390987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.390990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.391140 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.391647 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.391652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.391655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.392430 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:03.392583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:03.394691 (+ 2108us) server_negotiation.cc:299] Negotiation successful
0924 11:48:03.394924 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":670,"threads_started":1}
I20260924 11:48:03.395341 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.369126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.1.210.2:41043)
0924 11:48:03.369749 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:03.369771 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:03.371532 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:03.372589 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:03.372598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:03.373257 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:03.374669 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.374688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.376574 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.376581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.378923 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.378942 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.380048 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.380055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.380299 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.381574 (+ 1275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:03.381628 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:03.385846 (+ 4218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:03.390090 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.390099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.390115 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.390849 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.391269 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.391274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.391278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.391505 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.392572 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:03.392581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:03.393044 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 11:48:03.393391 (+ 347us) client_negotiation.cc:240] Negotiation successful
0924 11:48:03.394760 (+ 1369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":1454,"spinlock_wait_cycles":3968,"thread_start_us":293,"threads_started":1}
I20260924 11:48:03.396077 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.367826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.1.210.2:46669)
0924 11:48:03.368568 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:03.368613 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:03.371532 (+ 2919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:03.372236 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:03.372243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:03.373088 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:03.373963 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.373997 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.376013 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.376021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.379149 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.379162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.380652 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.380661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.380927 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.381583 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:03.381615 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:03.386580 (+ 4965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:03.390412 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.390419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.390423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.391057 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.391495 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.391500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.391504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.391704 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.393024 (+ 1320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:03.393030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:03.393200 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 11:48:03.393391 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 11:48:03.393945 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":44,"spinlock_wait_cycles":20864,"thread_start_us":323,"threads_started":1}
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.3@KRBTEST.COM
I20260924 11:48:03.397773 9128 heartbeater.cc:347] Connected to a master server at 127.1.210.61:37079
I20260924 11:48:03.397857 9126 heartbeater.cc:347] Connected to a master server at 127.1.210.60:43477
I20260924 11:48:03.398100 9128 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:03.398105 9126 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:03.399143 9128 heartbeater.cc:511] Master 127.1.210.61:37079 requested a full tablet report, sending...
I20260924 11:48:03.399169 9126 heartbeater.cc:511] Master 127.1.210.60:43477 requested a full tablet report, sending...
I20260924 11:48:03.400893 8608 ts_manager.cc:194] Registered new tserver with Master: 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2:42945)
I20260924 11:48:03.401115 8799 ts_manager.cc:194] Registered new tserver with Master: 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2:42945)
I20260924 11:48:03.403136 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.370671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:39727)
0924 11:48:03.371203 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:03.371224 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:03.371545 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:03.373013 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:03.373017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:03.373660 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:03.378820 (+ 5160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.378843 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.382262 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.382270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.385312 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.385331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.388264 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.388272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.390188 (+ 1916us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.390754 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:03.390781 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:03.394231 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:03.399057 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.399062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.399066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.399642 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.401146 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.401152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.401158 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.401495 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.402219 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:03.402223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:03.402355 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 11:48:03.402498 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 11:48:03.402772 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":247,"threads_started":1}
I20260924 11:48:03.403566 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.369998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:39727 (local address 127.1.210.62:43409)
0924 11:48:03.370909 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.370918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.371323 (+ 405us) server_negotiation.cc:407] Connection header received
0924 11:48:03.372758 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.372763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.372848 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:03.372944 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:03.379231 (+ 6287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.380206 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.385684 (+ 5478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.386168 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.386292 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.395183 (+ 8891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:03.395190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:03.395195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:03.395233 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:03.398747 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.400087 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.400092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.400095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.400245 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.401854 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.401858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.401860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.402100 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:03.402252 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:03.402596 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 11:48:03.402805 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":796,"threads_started":1}
I20260924 11:48:03.404289 9127 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43409
I20260924 11:48:03.404412 9127 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:03.404821 9127 heartbeater.cc:511] Master 127.1.210.62:43409 requested a full tablet report, sending...
I20260924 11:48:03.405721 8010 ts_manager.cc:194] Re-registered known tserver with Master: 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2:42945)
I20260924 11:48:03.405862 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.373756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.1.210.3:46459)
0924 11:48:03.385526 (+ 11770us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.385534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.385574 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:03.385638 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.385646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.385891 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:03.386049 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:03.387895 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.389263 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.390984 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.392219 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.392335 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.398160 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:03.398175 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:03.398192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:03.398238 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:03.401979 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.402635 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.402643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.402646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.402778 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.403623 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.403626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.403628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.404034 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:03.404307 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:03.405234 (+ 927us) server_negotiation.cc:299] Negotiation successful
0924 11:48:03.405483 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11599,"thread_start_us":12442,"threads_started":1}
I20260924 11:48:03.406519 8010 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:39727
I20260924 11:48:03.407816 8461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:03.491541 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8338
I20260924 11:48:03.507522 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:46459
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:43409
--webserver_port=35179
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:03.831859 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:03.832167 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:03.832343 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:48:03.838285 8980 heartbeater.cc:503] Master 127.1.210.62:43409 was elected leader, sending a full tablet report...
W20260924 11:48:03.840701 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:03.840818 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:03.841019 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:03.850730 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:46459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=35179
--tserver_master_addrs=127.1.210.62:43409,127.1.210.61:37079,127.1.210.60:43477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:03.852018 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:03.853350 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:03.866034 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:03.866125 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:03.866093 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:03.867328 9139 server_base.cc:1034] running on GCE node
I20260924 11:48:03.867918 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:03.868698 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:03.869886 9139 hybrid_clock.cc:648] HybridClock initialized: now 1790250483869848 us; error 31 us; skew 500 ppm
I20260924 11:48:03.870203 9139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:03.873384 9139 init.cc:377] Logged in from keytab as oryx/127.1.210.3@KRBTEST.COM (short username oryx)
I20260924 11:48:03.874795 9139 webserver.cc:460] Webserver started at http://127.1.210.3:35179/ using document root <none> and password file <none>
I20260924 11:48:03.875521 9139 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:03.875677 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:03.880653 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:48:03.883358 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:03.884675 9139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:48:03.884851 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "cfde18f86de144a59dde6f2436d80440"
format_stamp: "Formatted at 2026-09-24 11:48:00 on dist-test-slave-dd32"
server_key: "cae14bd8eab8c5769141c37b12f9e1cb"
server_key_iv: "6161ad613ba5db7936d90008a8a38b4a"
server_key_version: "encryptionkey@0"
I20260924 11:48:03.885910 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:03.905797 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:03.909512 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:03.909771 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:03.910653 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:03.911937 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:03.912056 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:03.912200 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:03.912286 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:03.957276 9139 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:46459
I20260924 11:48:03.957370 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:46459 every 8 connection(s)
I20260924 11:48:03.959074 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:03.967090 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 9139
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.60@KRBTEST.COM
I20260924 11:48:03.990350 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.961806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:37009 (local address 127.1.210.60:43477)
0924 11:48:03.962236 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.962242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.963835 (+ 1593us) server_negotiation.cc:407] Connection header received
0924 11:48:03.967202 (+ 3367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.967212 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.967328 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:03.967437 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:03.970260 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.971793 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.975438 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.976051 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.976158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.982064 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:03.982086 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:03.982094 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:03.982145 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:03.986219 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.987255 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.987264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.987271 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.987423 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:03.987964 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:03.987972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:03.987978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:03.988387 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:03.988530 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:03.989639 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0924 11:48:03.989851 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":223,"threads_started":1}
I20260924 11:48:03.992676 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.962798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.1.210.3:37009)
0924 11:48:03.963625 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:03.963671 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:03.966785 (+ 3114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:03.968083 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:03.968099 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:03.968917 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:03.970008 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.970038 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.972184 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.972192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.975246 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.975266 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.976486 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.976497 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.976869 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.977802 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:03.977847 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:03.981772 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:03.986399 (+ 4627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.986411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.986432 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.987106 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.987580 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:03.987587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:03.987593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:03.987831 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:03.988617 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:03.988626 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:03.989150 (+ 524us) client_negotiation.cc:769] Sending connection context
0924 11:48:03.990833 (+ 1683us) client_negotiation.cc:240] Negotiation successful
0924 11:48:03.991361 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":348,"threads_started":1}
I20260924 11:48:03.995327 9267 heartbeater.cc:347] Connected to a master server at 127.1.210.60:43477
I20260924 11:48:03.995605 9267 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:03.996853 9267 heartbeater.cc:511] Master 127.1.210.60:43477 requested a full tablet report, sending...
Sep 24 11:48:03 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.61@KRBTEST.COM
I20260924 11:48:03.998793 8799 ts_manager.cc:194] Registered new tserver with Master: cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
Sep 24 11:48:04 dist-test-slave-dd32 krb5kdc[7965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:04.011447 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.982316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.1.210.3:45251)
0924 11:48:03.988617 (+ 6301us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:03.988640 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:03.988734 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:03.990291 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:03.990296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:03.991031 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:03.991294 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.991306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.992723 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.992728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.993852 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:03.993863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.994948 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.994953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:03.995119 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.995799 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:03.995819 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:03.998979 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:04.004065 (+ 5086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:04.004074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:04.004081 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:04.004839 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:04.007887 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:04.008007 (+ 120us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:04.008015 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:04.008218 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:04.009949 (+ 1731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:04.009959 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:04.010183 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.010380 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.010785 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6165,"thread_start_us":6478,"threads_started":1}
I20260924 11:48:04.012562 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.976326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:45251 (local address 127.1.210.61:37079)
0924 11:48:03.989809 (+ 13483us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.989815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.989834 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:48:03.989888 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.989894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.989992 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:03.990093 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:03.991465 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.992527 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.993990 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:03.994583 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:03.994704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:03.999251 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:03.999263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:03.999297 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:03.999373 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.003815 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.007469 (+ 3654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.007479 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.007485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.007701 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.009249 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.009257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.009263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.009798 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.010020 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.011830 (+ 1810us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.012076 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13354,"thread_start_us":14454,"threads_started":1}
I20260924 11:48:04.015331 9269 heartbeater.cc:347] Connected to a master server at 127.1.210.61:37079
I20260924 11:48:04.015949 9269 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:04.016448 9269 heartbeater.cc:511] Master 127.1.210.61:37079 requested a full tablet report, sending...
I20260924 11:48:04.017874 8608 ts_manager.cc:194] Registered new tserver with Master: cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:04.022825 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.964253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.3:58047)
0924 11:48:03.975634 (+ 11381us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:03.975652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:03.975751 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:04.000052 (+ 24301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:04.000056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:04.000626 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:04.000830 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.000838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.002156 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.002163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.003101 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.003114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.004292 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.004302 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.004580 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.005462 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:04.005497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:04.009208 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:04.019151 (+ 9943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:04.019156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:04.019162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:04.020615 (+ 1453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:04.021003 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:04.021008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:04.021011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:04.021193 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:04.021862 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:04.021866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:04.022013 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.022157 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.022436 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11257,"thread_start_us":11557,"threads_started":1}
I20260924 11:48:04.023680 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:03.964431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:58047 (local address 127.1.210.62:43409)
0924 11:48:03.999796 (+ 35365us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:03.999801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:03.999815 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:48:03.999859 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:03.999862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:03.999916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.000034 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.000977 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.002017 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.003271 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.003997 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.004957 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.009526 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.009537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.009546 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.009599 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.017503 (+ 7904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.020748 (+ 3245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.020754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.020757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.020884 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.021323 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.021328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.021331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.021637 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.021795 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.023108 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.023338 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35265,"thread_start_us":33682,"threads_started":1}
I20260924 11:48:04.024897 9268 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43409
I20260924 11:48:04.025023 9268 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:04.025452 9268 heartbeater.cc:511] Master 127.1.210.62:43409 requested a full tablet report, sending...
I20260924 11:48:04.026523 8010 ts_manager.cc:194] Re-registered known tserver with Master: cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:04.027664 8010 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:58047
I20260924 11:48:04.033775 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.003415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.1.210.62:43409)
0924 11:48:04.011598 (+ 8183us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.011605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.011627 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:48:04.011675 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.011682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.011767 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.011897 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.014598 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.016558 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.019565 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.020188 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.020303 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.024986 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.024992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.024996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.025031 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.028864 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.030057 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.030066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.030070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.030219 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.032148 (+ 1929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.032154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.032158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.032490 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.032750 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.033113 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.033357 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8026,"thread_start_us":7890,"threads_started":1}
I20260924 11:48:04.034917 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.010707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.1.210.61:37079)
0924 11:48:04.011185 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.011191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.013445 (+ 2254us) server_negotiation.cc:407] Connection header received
0924 11:48:04.013727 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.013733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.013797 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.013867 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.019111 (+ 5244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.020297 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.022342 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.022993 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.023080 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.027302 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.027309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.027313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.027346 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.031603 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.032678 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.032684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.032687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.032829 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.033271 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.033276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.033279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.033565 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.033735 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.034244 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.034454 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260924 11:48:04.037845 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.013522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.1.210.60:43477)
0924 11:48:04.013711 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.013716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.019062 (+ 5346us) server_negotiation.cc:407] Connection header received
0924 11:48:04.019344 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.019351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.019465 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.019545 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.022197 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.023798 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.026320 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.027100 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.027229 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.030956 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.030963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.030969 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.031020 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.034909 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.035769 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.035774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.035778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.035985 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.036391 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.036397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.036400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.036667 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.036923 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.037320 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.037569 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:48:04.039698 8010 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:04.046943 8010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:04.089049 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.065054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.1.210.1:44009)
0924 11:48:04.078517 (+ 13463us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.078526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.078562 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:48:04.078648 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.078654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.078856 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.079065 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:04.079725 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.081519 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.084958 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.087152 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.087286 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.087571 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.088385 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.088614 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13253,"thread_start_us":13138,"threads_started":1}
I20260924 11:48:04.090307 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.064806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:44009 (local address 127.0.0.1:40916)
0924 11:48:04.077316 (+ 12510us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:04.077336 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:04.077459 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:04.079042 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:04.079047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:04.079072 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:04.079499 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.079516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.081736 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.081740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.084737 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.084757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.087812 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.087821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.088127 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.088180 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.089719 (+ 1539us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.089867 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12379,"mutex_wait_us":1,"thread_start_us":14589,"threads_started":1}
I20260924 11:48:04.093132 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.065226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:46459 (local address 127.0.0.1:33870)
0924 11:48:04.079711 (+ 14485us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:04.079727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:04.079828 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:04.082897 (+ 3069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:04.082906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:04.082941 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:04.083516 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.083532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.086415 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.086424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.089314 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.089334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.092334 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.092341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.092514 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.092551 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.092654 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.092774 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14333,"mutex_wait_us":29,"thread_start_us":10923,"threads_started":1}
I20260924 11:48:04.093655 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.065479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.1.210.3:46459)
0924 11:48:04.081910 (+ 16431us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.081923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.081963 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:04.082075 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.082085 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.082406 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.083775 (+ 1369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:04.084110 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.086224 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.089593 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.091729 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.092277 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.092477 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.092717 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.092860 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16203,"thread_start_us":17106,"threads_started":1}
I20260924 11:48:04.096371 9202 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d2f12e9fe4f429fab13ef3a5fb176 (DEFAULT_TABLE table=test-table [id=f83c4959f2034b92b2fd6a8d81ceb508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:04.096415 8914 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d2f12e9fe4f429fab13ef3a5fb176 (DEFAULT_TABLE table=test-table [id=f83c4959f2034b92b2fd6a8d81ceb508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:04.097712 9202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d2f12e9fe4f429fab13ef3a5fb176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:04.097816 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d2f12e9fe4f429fab13ef3a5fb176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:04.097879 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.080125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:42945 (local address 127.0.0.1:51892)
0924 11:48:04.090690 (+ 10565us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:04.090706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:04.090792 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:04.091728 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:04.091733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:04.091756 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:04.091935 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.091944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.093848 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.093853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.095523 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.095536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.097184 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.097191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.097369 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.097409 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.097479 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.097604 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10477,"thread_start_us":11868,"threads_started":1}
I20260924 11:48:04.098778 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.080116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.1.210.2:42945)
0924 11:48:04.091280 (+ 11164us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.091293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.091314 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:48:04.091383 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.091387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.091553 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.091747 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:04.092100 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.093681 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.095715 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.096974 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.098110 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.098296 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.098362 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.098510 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11006,"thread_start_us":10911,"threads_started":1}
I20260924 11:48:04.102384 9061 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d2f12e9fe4f429fab13ef3a5fb176 (DEFAULT_TABLE table=test-table [id=f83c4959f2034b92b2fd6a8d81ceb508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:04.103627 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d2f12e9fe4f429fab13ef3a5fb176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:04.109249 9299 tablet_bootstrap.cc:492] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Bootstrap starting.
I20260924 11:48:04.112205 9299 tablet_bootstrap.cc:654] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:04.113412 9299 log.cc:824] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:04.118054 9301 tablet_bootstrap.cc:492] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Bootstrap starting.
I20260924 11:48:04.120754 9301 tablet_bootstrap.cc:654] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:04.121907 9301 log.cc:824] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:04.123157 9299 tablet_bootstrap.cc:492] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: No bootstrap required, opened a new log
I20260924 11:48:04.123430 9299 ts_tablet_manager.cc:1397] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260924 11:48:04.124505 9301 tablet_bootstrap.cc:492] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: No bootstrap required, opened a new log
I20260924 11:48:04.124784 9301 ts_tablet_manager.cc:1397] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.006s
I20260924 11:48:04.129249 9302 tablet_bootstrap.cc:492] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Bootstrap starting.
I20260924 11:48:04.132721 9299 raft_consensus.cc:348] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.133195 9299 raft_consensus.cc:374] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:04.133306 9299 raft_consensus.cc:729] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfde18f86de144a59dde6f2436d80440, State: Initialized, Role: FOLLOWER
I20260924 11:48:04.133363 9302 tablet_bootstrap.cc:654] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:04.133278 9301 raft_consensus.cc:348] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.133683 9301 raft_consensus.cc:374] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:04.133780 9301 raft_consensus.cc:729] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3ce6361a0c481eaf81bc1b655b8a53, State: Initialized, Role: FOLLOWER
I20260924 11:48:04.133975 9299 consensus_queue.cc:260] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.134400 9301 consensus_queue.cc:260] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.135294 9302 log.cc:824] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:04.142241 9301 ts_tablet_manager.cc:1428] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260924 11:48:04.143451 9302 tablet_bootstrap.cc:492] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: No bootstrap required, opened a new log
I20260924 11:48:04.143833 9302 ts_tablet_manager.cc:1397] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.010s
I20260924 11:48:04.151716 9299 ts_tablet_manager.cc:1428] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Time spent starting tablet: real 0.028s user 0.010s sys 0.018s
I20260924 11:48:04.152066 9268 heartbeater.cc:503] Master 127.1.210.62:43409 was elected leader, sending a full tablet report...
I20260924 11:48:04.157572 9302 raft_consensus.cc:348] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.158068 9302 raft_consensus.cc:374] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:04.158172 9302 raft_consensus.cc:729] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d46a63fa478b82cead56cfaa6259, State: Initialized, Role: FOLLOWER
I20260924 11:48:04.158746 9302 consensus_queue.cc:260] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.159700 9127 heartbeater.cc:503] Master 127.1.210.62:43409 was elected leader, sending a full tablet report...
I20260924 11:48:04.160356 9302 ts_tablet_manager.cc:1428] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260924 11:48:04.219367 9270 tablet.cc:2369] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:48:04.241293 9307 raft_consensus.cc:482] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:04.241559 9307 raft_consensus.cc:504] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.243664 9307 leader_election.cc:288] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009), cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:04.250708 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.243978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:39897 (local address 127.1.210.1:44009)
0924 11:48:04.244199 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.244206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.245128 (+ 922us) server_negotiation.cc:407] Connection header received
0924 11:48:04.245441 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.245447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.245532 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.245604 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:04.246317 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.247002 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.248750 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.249717 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.249876 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.249982 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.250289 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.250437 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:48:04.251082 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.244140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:46459 (local address 127.1.210.2:33835)
0924 11:48:04.245035 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:04.245049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:04.245284 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:04.245639 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:04.245642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:04.245660 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:04.245929 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.245937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.247024 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.247029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.248540 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.248551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.249751 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.249759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.250019 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.250056 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.250139 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.250262 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":3,"thread_start_us":652,"threads_started":1}
I20260924 11:48:04.251096 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.244377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:33835 (local address 127.1.210.3:46459)
0924 11:48:04.244620 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.244627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.245174 (+ 547us) server_negotiation.cc:407] Connection header received
0924 11:48:04.245442 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.245447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.245522 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.245613 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:04.246115 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.246893 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.248773 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.249551 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.249648 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.249721 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.250181 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.250318 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:48:04.251389 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.244028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:44009 (local address 127.1.210.2:39897)
0924 11:48:04.244965 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:04.244985 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:04.245244 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:04.245807 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:04.245810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:04.245827 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:04.246171 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.246183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.247156 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.247159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.248571 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:04.248582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.249965 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.249971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:04.250103 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.250156 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:48:04.250247 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 11:48:04.250355 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":24,"thread_start_us":525,"threads_started":1}
I20260924 11:48:04.252228 9222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d2f12e9fe4f429fab13ef3a5fb176" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde18f86de144a59dde6f2436d80440" is_pre_election: true
I20260924 11:48:04.252318 8934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d2f12e9fe4f429fab13ef3a5fb176" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" is_pre_election: true
I20260924 11:48:04.252734 9222 raft_consensus.cc:2454] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 0.
I20260924 11:48:04.252823 8934 raft_consensus.cc:2454] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 0.
I20260924 11:48:04.253461 9017 leader_election.cc:302] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f2d46a63fa478b82cead56cfaa6259, cfde18f86de144a59dde6f2436d80440; no voters:
I20260924 11:48:04.253867 9307 raft_consensus.cc:2781] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:04.254071 9307 raft_consensus.cc:482] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:04.254204 9307 raft_consensus.cc:3037] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:04.256480 9307 raft_consensus.cc:504] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.257378 9307 leader_election.cc:288] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 election: Requested vote from peers 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009), cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:04.257694 8934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d2f12e9fe4f429fab13ef3a5fb176" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53"
I20260924 11:48:04.257818 9222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d2f12e9fe4f429fab13ef3a5fb176" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde18f86de144a59dde6f2436d80440"
I20260924 11:48:04.257893 8934 raft_consensus.cc:3037] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:04.258044 9222 raft_consensus.cc:3037] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:04.260116 9222 raft_consensus.cc:2454] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 1.
I20260924 11:48:04.260161 8934 raft_consensus.cc:2454] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 1.
I20260924 11:48:04.260468 9017 leader_election.cc:302] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f2d46a63fa478b82cead56cfaa6259, cfde18f86de144a59dde6f2436d80440; no voters:
I20260924 11:48:04.260774 9307 raft_consensus.cc:2781] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:04.261278 9307 raft_consensus.cc:686] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 LEADER]: Becoming Leader. State: Replica: 90f2d46a63fa478b82cead56cfaa6259, State: Running, Role: LEADER
I20260924 11:48:04.261988 9307 consensus_queue.cc:237] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:04.267585 8009 catalog_manager.cc:5495] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "90f2d46a63fa478b82cead56cfaa6259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } health_report { overall_health: HEALTHY } } }
W20260924 11:48:04.305907 8982 tablet.cc:2369] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:48:04.325436 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.320034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.1.210.2:42945)
0924 11:48:04.320240 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.320247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.320351 (+ 104us) server_negotiation.cc:407] Connection header received
0924 11:48:04.320608 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.320612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.320688 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.320838 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:04.321248 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.321939 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.323353 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.323753 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.323890 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.324350 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:04.324847 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:04.325010 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.325065 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.325170 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:48:04.333777 9222 raft_consensus.cc:1261] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Refusing update from remote peer 90f2d46a63fa478b82cead56cfaa6259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:04.333781 8934 raft_consensus.cc:1261] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Refusing update from remote peer 90f2d46a63fa478b82cead56cfaa6259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:04.334764 9310 consensus_queue.cc:1035] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:04.335111 9307 consensus_queue.cc:1035] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:04.347416 9314 mvcc.cc:205] Tried to move back new op lower bound from 7332865983822123008 to 7332865983540895744. Current Snapshot: MvccSnapshot[applied={T|T < 7332865983822123008}]
I20260924 11:48:04.370175 8009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:04.370435 8009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:04.377301 8009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aa51beb1207440099d8e59611229b44: Sending DeleteTablet for 3 replicas of tablet cc0d2f12e9fe4f429fab13ef3a5fb176
I20260924 11:48:04.378362 8914 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d2f12e9fe4f429fab13ef3a5fb176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:04 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:40916
I20260924 11:48:04.378517 9061 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d2f12e9fe4f429fab13ef3a5fb176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:04 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260924 11:48:04.378804 9202 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d2f12e9fe4f429fab13ef3a5fb176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:04 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:33870
I20260924 11:48:04.397810 9326 tablet_replica.cc:330] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: stopping tablet replica
I20260924 11:48:04.398408 9326 raft_consensus.cc:2229] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:04.399595 9326 raft_consensus.cc:2258] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:04.401866 9326 ts_tablet_manager.cc:1905] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:04.408149 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 7978
I20260924 11:48:04.409512 8040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:04.412608 9327 tablet_replica.cc:330] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: stopping tablet replica
I20260924 11:48:04.413386 9327 raft_consensus.cc:2229] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:04.414148 9327 raft_consensus.cc:2258] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:04.416121 9325 tablet_replica.cc:330] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: stopping tablet replica
I20260924 11:48:04.416467 9325 raft_consensus.cc:2229] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:04.416879 9325 raft_consensus.cc:2258] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:04.417973 9326 ts_tablet_manager.cc:1918] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:04.418166 9326 log.cc:1196] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/cc0d2f12e9fe4f429fab13ef3a5fb176
I20260924 11:48:04.418723 9326 ts_tablet_manager.cc:1939] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 90f2d46a63fa478b82cead56cfaa6259: Deleting consensus metadata
I20260924 11:48:04.418951 9325 ts_tablet_manager.cc:1905] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:04.421692 9327 ts_tablet_manager.cc:1905] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:04.425760 9325 ts_tablet_manager.cc:1918] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:04.425920 9325 log.cc:1196] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/cc0d2f12e9fe4f429fab13ef3a5fb176
I20260924 11:48:04.426435 9325 ts_tablet_manager.cc:1939] T cc0d2f12e9fe4f429fab13ef3a5fb176 P 4c3ce6361a0c481eaf81bc1b655b8a53: Deleting consensus metadata
I20260924 11:48:04.432526 9327 ts_tablet_manager.cc:1918] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:04.432718 9327 log.cc:1196] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/cc0d2f12e9fe4f429fab13ef3a5fb176
I20260924 11:48:04.433264 9327 ts_tablet_manager.cc:1939] T cc0d2f12e9fe4f429fab13ef3a5fb176 P cfde18f86de144a59dde6f2436d80440: Deleting consensus metadata
I20260924 11:48:04.440147 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.381762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.1.210.61:37079)
0924 11:48:04.382014 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.382023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.406238 (+ 24215us) server_negotiation.cc:407] Connection header received
0924 11:48:04.409392 (+ 3154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.409402 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.409520 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.409627 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.418233 (+ 8606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.419291 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.423245 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.423895 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.423995 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.434002 (+ 10007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.434010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.434015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.434058 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.437205 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.437966 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.437971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.437975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.438106 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.438550 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.438555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.438560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.438827 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.439093 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.439420 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.439648 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:48:04.460309 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.406878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.1.210.60:43477)
0924 11:48:04.407040 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.407046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.440004 (+ 32958us) server_negotiation.cc:407] Connection header received
0924 11:48:04.440166 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.440172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.440256 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.440367 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.446403 (+ 6036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.447538 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.449417 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.450097 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.450216 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.453671 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.453678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.453683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.453721 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.457461 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.458247 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.458252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.458256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.458405 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.458820 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.458826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.458830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.459165 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.459418 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.459732 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.459935 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:48:04.557307 7997 catalog_manager.cc:4841] TS cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459): tablet cc0d2f12e9fe4f429fab13ef3a5fb176 (table test-table [id=f83c4959f2034b92b2fd6a8d81ceb508]) successfully deleted
I20260924 11:48:04.558200 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 7978
W20260924 11:48:04.579571 8869 connection.cc:535] server connection from 127.0.0.1:40916 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:48:04.601460 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.583971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.1.210.61:37079)
0924 11:48:04.584175 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.584184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.584350 (+ 166us) server_negotiation.cc:407] Connection header received
0924 11:48:04.584619 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.584625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.584712 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.584821 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.587068 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.588281 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.590430 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.591106 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.591199 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.594934 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.594940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.594945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.594982 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.598470 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.599214 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.599219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.599223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.599347 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.599762 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.599767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.599771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.600116 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.600359 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.600862 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.601068 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:48:04.602660 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:04.584877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.1.210.60:43477)
0924 11:48:04.585191 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:04.585197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:04.585237 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:04.585420 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:04.585426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:04.585507 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:04.585656 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:04.587917 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.589034 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.591082 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:04.591770 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:04.591870 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:04.595459 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:04.595466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:04.595471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:04.595509 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:04.599268 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.600105 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.600111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.600116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.600246 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:04.600722 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:04.600727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:04.600731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:04.601107 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:04.601318 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:04.602136 (+ 818us) server_negotiation.cc:299] Negotiation successful
0924 11:48:04.602343 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:48:05.355293 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.354455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.3:51559)
0924 11:48:05.354967 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.355107 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
W20260924 11:48:05.355649 9268 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:43409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
I20260924 11:48:05.360826 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.360234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:57001)
0924 11:48:05.360575 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.360675 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":179,"threads_started":1}
W20260924 11:48:05.361193 9127 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:43409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
I20260924 11:48:05.363876 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.363233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:53637)
0924 11:48:05.363602 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.363742 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":191,"threads_started":1}
W20260924 11:48:05.364192 8980 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:43409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
I20260924 11:48:05.877884 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5aa51beb1207440099d8e59611229b44)
I20260924 11:48:05.878176 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } }
I20260924 11:48:05.878530 9340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5aa51beb1207440099d8e59611229b44)
I20260924 11:48:05.878751 9340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } }
I20260924 11:48:05.879696 9339 leader_election.cc:288] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409), 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60:43477)
I20260924 11:48:05.879712 9340 leader_election.cc:288] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409), ebb5374cc6164b31acbed1081a4843e7 (127.1.210.61:37079)
I20260924 11:48:05.880720 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.879787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57216)
0924 11:48:05.880285 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.880466 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260924 11:48:05.881013 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.879829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57210)
0924 11:48:05.880460 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.880744 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":349,"threads_started":1}
W20260924 11:48:05.882311 8592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
W20260924 11:48:05.882376 8782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
I20260924 11:48:05.891460 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.880416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:43477 (local address 127.0.0.1:52416)
0924 11:48:05.881158 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.881181 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:05.881343 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:05.882717 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:05.882724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:05.882750 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:05.883043 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.883058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.885453 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.885462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.888197 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.888216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.890124 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.890130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.890481 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.890547 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 11:48:05.890692 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 11:48:05.890903 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":25,"thread_start_us":1581,"threads_started":1}
I20260924 11:48:05.891909 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.881851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.1.210.60:43477)
0924 11:48:05.882391 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:05.882397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:05.882417 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:48:05.882487 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:05.882492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:05.882557 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:05.883226 (+ 669us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:05.883601 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.885258 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.888480 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.889929 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.890300 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.890405 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:05.891423 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0924 11:48:05.891546 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
I20260924 11:48:05.893054 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebb5374cc6164b31acbed1081a4843e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34e8b054e27145cda2ad94ba4c0f1527" is_pre_election: true
I20260924 11:48:05.893375 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.880560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:37079 (local address 127.0.0.1:39714)
0924 11:48:05.881411 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.881432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:05.881613 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:05.882731 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:05.882739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:05.882775 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:05.883162 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.883179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.885805 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.885815 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.888667 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.888689 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.892358 (+ 3669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.892373 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.892602 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.892655 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:48:05.892743 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:48:05.892889 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":1039,"threads_started":1}
I20260924 11:48:05.893481 8809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb5374cc6164b31acbed1081a4843e7 in term 1.
I20260924 11:48:05.893664 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.893008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57232)
0924 11:48:05.893231 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.893422 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:48:05.893689 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.893338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57240)
0924 11:48:05.893473 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.893553 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 11:48:05.893896 8782 leader_election.cc:334] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409): Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
W20260924 11:48:05.893929 8592 leader_election.cc:334] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409): Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
I20260924 11:48:05.894296 8592 leader_election.cc:302] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e8b054e27145cda2ad94ba4c0f1527, ebb5374cc6164b31acbed1081a4843e7; no voters: 5aa51beb1207440099d8e59611229b44
I20260924 11:48:05.894608 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.880873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.1.210.61:37079)
0924 11:48:05.882367 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:05.882375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:05.882402 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:48:05.882470 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:05.882475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:05.882560 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:05.883351 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:05.883723 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.885519 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.889015 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.892106 (+ 3091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.892832 (+ 726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.894095 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:05.894145 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 11:48:05.894260 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":1377,"thread_start_us":1052,"threads_started":1}
I20260924 11:48:05.894682 9339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 11:48:05.894826 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5aa51beb1207440099d8e59611229b44)
I20260924 11:48:05.894927 9339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:48:05.895104 8618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34e8b054e27145cda2ad94ba4c0f1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ebb5374cc6164b31acbed1081a4843e7" is_pre_election: true
I20260924 11:48:05.897009 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } }
I20260924 11:48:05.897433 8618 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 34e8b054e27145cda2ad94ba4c0f1527 in current term 2: Already voted for candidate ebb5374cc6164b31acbed1081a4843e7 in this term.
I20260924 11:48:05.897746 9339 leader_election.cc:288] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [CANDIDATE]: Term 2 election: Requested vote from peers 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409), 34e8b054e27145cda2ad94ba4c0f1527 (127.1.210.60:43477)
I20260924 11:48:05.898212 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.897818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57244)
0924 11:48:05.897955 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.898062 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:48:05.898350 8782 leader_election.cc:302] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e8b054e27145cda2ad94ba4c0f1527; no voters: 5aa51beb1207440099d8e59611229b44, ebb5374cc6164b31acbed1081a4843e7
I20260924 11:48:05.898329 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebb5374cc6164b31acbed1081a4843e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34e8b054e27145cda2ad94ba4c0f1527"
I20260924 11:48:05.898648 8809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:48:05.899276 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.898945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57260)
0924 11:48:05.899100 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.899169 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 11:48:05.899415 8592 leader_election.cc:334] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409): Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
I20260924 11:48:05.900708 8809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb5374cc6164b31acbed1081a4843e7 in term 2.
I20260924 11:48:05.900899 9340 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:48:05.901144 8592 leader_election.cc:302] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e8b054e27145cda2ad94ba4c0f1527, ebb5374cc6164b31acbed1081a4843e7; no voters: 5aa51beb1207440099d8e59611229b44
I20260924 11:48:05.901443 9339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:48:05.901880 9339 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [term 2 LEADER]: Becoming Leader. State: Replica: ebb5374cc6164b31acbed1081a4843e7, State: Running, Role: LEADER
I20260924 11:48:05.902275 9339 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } }
I20260924 11:48:05.903901 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb5374cc6164b31acbed1081a4843e7. Latest consensus state: current_term: 2 leader_uuid: "ebb5374cc6164b31acbed1081a4843e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:05.904202 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:05.904821 9351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:05.908160 9351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:05.908927 9351 catalog_manager.cc:1259] Loaded cluster ID: 45533faa9e354fc9aa415961c72bdb5e
I20260924 11:48:05.909035 9351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:05.910161 9351 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:05.911001 9351 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Loaded TSK: 0
I20260924 11:48:05.911818 9351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:48:05.945204 8607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:05.952143 8607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:05.956018 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.955678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57276)
0924 11:48:05.955812 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.955890 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 11:48:05.956504 8809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [term 2 FOLLOWER]: Refusing update from remote peer ebb5374cc6164b31acbed1081a4843e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 11:48:05.957238 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.956694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57282)
0924 11:48:05.956902 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.957039 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260924 11:48:05.957446 8592 consensus_peers.cc:487] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 -> Peer 5aa51beb1207440099d8e59611229b44 (127.1.210.62:43409): Couldn't send request to peer 5aa51beb1207440099d8e59611229b44. Status: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 11:48:05.957504 9349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 11:48:05.960263 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb5374cc6164b31acbed1081a4843e7. Latest consensus state: current_term: 2 leader_uuid: "ebb5374cc6164b31acbed1081a4843e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:05.960572 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:48:05.962409 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ebb5374cc6164b31acbed1081a4843e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:05.962499 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ebb5374cc6164b31acbed1081a4843e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5aa51beb1207440099d8e59611229b44" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43409 } } peers { permanent_uuid: "ebb5374cc6164b31acbed1081a4843e7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 37079 } attrs { promote: false } } peers { permanent_uuid: "34e8b054e27145cda2ad94ba4c0f1527" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 43477 } attrs { promote: false } } }
I20260924 11:48:05.962694 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:05.962772 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e8b054e27145cda2ad94ba4c0f1527 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:48:05.979676 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.970248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:44009 (local address 127.0.0.1:40918)
0924 11:48:05.970385 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.970405 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:05.970511 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:05.971371 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:05.971379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:05.971416 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:05.971704 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.971723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.973770 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.973779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.976381 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.976400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.978620 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.978630 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.978905 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.978965 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 11:48:05.979046 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 11:48:05.979255 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 11:48:05.981433 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.970497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.1.210.1:44009)
0924 11:48:05.971018 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:05.971024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:05.971043 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:48:05.971121 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:05.971127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:05.971198 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:05.971898 (+ 700us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:05.972124 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.973577 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.976668 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.978378 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.980588 (+ 2210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.980737 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:05.980801 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 11:48:05.980987 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":302,"threads_started":1}
I20260924 11:48:05.985487 8914 tablet_service.cc:1467] Processing CreateTablet for tablet 5b074f3288b44521b38f7a89413783b9 (DEFAULT_TABLE table=test-table [id=786a997131eb4022b8680a83e8ccf707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:05.985930 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b074f3288b44521b38f7a89413783b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:05.991881 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.971166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.1.210.2:42945)
0924 11:48:05.986411 (+ 15245us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:05.986418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:05.986438 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:48:05.986499 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:05.986528 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:05.986607 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:05.986710 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:05.987118 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.988245 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.990251 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.990956 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.991105 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.991211 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:05.991482 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 11:48:05.991610 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15116,"thread_start_us":15029,"threads_started":1}
I20260924 11:48:05.991950 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.971044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:42945 (local address 127.0.0.1:51922)
0924 11:48:05.979958 (+ 8914us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.979980 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:05.980095 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:05.986747 (+ 6652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:05.986751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:05.986774 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:05.986965 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.986974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.988376 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.988383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.990068 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.990080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.991116 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.991122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.991325 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.991347 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:48:05.991460 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 11:48:05.991587 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8792,"mutex_wait_us":28,"thread_start_us":13400,"threads_started":1}
I20260924 11:48:05.992262 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.970610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:46459 (local address 127.0.0.1:33878)
0924 11:48:05.970980 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:05.970997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:05.971102 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:05.985935 (+ 14833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:05.985940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:05.985970 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:05.986260 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.986271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.987612 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.987624 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.989869 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:05.989884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.991496 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.991502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:05.991681 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.991713 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:48:05.991777 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 11:48:05.991896 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":107}
I20260924 11:48:05.993140 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:05.971322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.1.210.3:46459)
0924 11:48:05.985541 (+ 14219us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:05.985550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:05.985574 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:48:05.985656 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:05.985664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:05.985771 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:05.985902 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:05.986452 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.987481 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.990052 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:05.991260 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:05.992316 (+ 1056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:05.992440 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:05.992502 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 11:48:05.992706 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14066,"thread_start_us":13949,"threads_started":1}
I20260924 11:48:05.993023 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 5b074f3288b44521b38f7a89413783b9 (DEFAULT_TABLE table=test-table [id=786a997131eb4022b8680a83e8ccf707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:05.993522 9362 tablet_bootstrap.cc:492] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Bootstrap starting.
I20260924 11:48:05.993520 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b074f3288b44521b38f7a89413783b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:05.993818 9202 tablet_service.cc:1467] Processing CreateTablet for tablet 5b074f3288b44521b38f7a89413783b9 (DEFAULT_TABLE table=test-table [id=786a997131eb4022b8680a83e8ccf707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:05.994168 9202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b074f3288b44521b38f7a89413783b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:05.995211 9362 tablet_bootstrap.cc:654] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:05.997344 9362 tablet_bootstrap.cc:492] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: No bootstrap required, opened a new log
I20260924 11:48:05.997490 9362 ts_tablet_manager.cc:1397] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:48:05.998263 9362 raft_consensus.cc:348] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:05.998515 9362 raft_consensus.cc:374] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:05.998625 9362 raft_consensus.cc:729] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3ce6361a0c481eaf81bc1b655b8a53, State: Initialized, Role: FOLLOWER
I20260924 11:48:05.998890 9362 consensus_queue.cc:260] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:05.999748 9362 ts_tablet_manager.cc:1428] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:06.005558 8981 heartbeater.cc:503] Master 127.1.210.61:37079 was elected leader, sending a full tablet report...
I20260924 11:48:06.007791 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.000440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:58517)
0924 11:48:06.007468 (+ 7028us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.007585 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":790,"threads_started":1}
I20260924 11:48:06.008308 9365 tablet_bootstrap.cc:492] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Bootstrap starting.
I20260924 11:48:06.009654 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.008467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:44355)
0924 11:48:06.009125 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.009299 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":303,"threads_started":1}
I20260924 11:48:06.009958 9365 tablet_bootstrap.cc:654] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:06.011252 9366 tablet_bootstrap.cc:492] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Bootstrap starting.
I20260924 11:48:06.012816 9366 tablet_bootstrap.cc:654] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:06.014587 9365 tablet_bootstrap.cc:492] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: No bootstrap required, opened a new log
I20260924 11:48:06.014732 9365 ts_tablet_manager.cc:1397] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260924 11:48:06.014812 9366 tablet_bootstrap.cc:492] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: No bootstrap required, opened a new log
I20260924 11:48:06.014946 9366 ts_tablet_manager.cc:1397] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:48:06.015463 9365 raft_consensus.cc:348] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.015635 9365 raft_consensus.cc:374] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:06.015538 9366 raft_consensus.cc:348] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.015734 9365 raft_consensus.cc:729] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfde18f86de144a59dde6f2436d80440, State: Initialized, Role: FOLLOWER
I20260924 11:48:06.015766 9366 raft_consensus.cc:374] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:06.015868 9366 raft_consensus.cc:729] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d46a63fa478b82cead56cfaa6259, State: Initialized, Role: FOLLOWER
I20260924 11:48:06.015985 9365 consensus_queue.cc:260] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.016083 9366 consensus_queue.cc:260] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.016745 9365 ts_tablet_manager.cc:1428] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:06.016850 9366 ts_tablet_manager.cc:1428] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:06.018708 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.017516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.3:57109)
0924 11:48:06.018177 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.018384 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":263,"threads_started":1}
I20260924 11:48:06.020310 9269 heartbeater.cc:503] Master 127.1.210.61:37079 was elected leader, sending a full tablet report...
I20260924 11:48:06.022148 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.017455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:58709)
0924 11:48:06.021601 (+ 4146us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.021739 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3999,"thread_start_us":3888,"threads_started":1}
I20260924 11:48:06.023734 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.023387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:41991)
0924 11:48:06.023526 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.023601 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:48:06.023809 9128 heartbeater.cc:503] Master 127.1.210.61:37079 was elected leader, sending a full tablet report...
I20260924 11:48:06.039371 9372 raft_consensus.cc:482] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:06.039573 9372 raft_consensus.cc:504] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.040246 9372 leader_election.cc:288] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009), cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:06.040702 8934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b074f3288b44521b38f7a89413783b9" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" is_pre_election: true
I20260924 11:48:06.040767 9222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b074f3288b44521b38f7a89413783b9" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde18f86de144a59dde6f2436d80440" is_pre_election: true
I20260924 11:48:06.040973 9222 raft_consensus.cc:2454] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 0.
I20260924 11:48:06.040963 8934 raft_consensus.cc:2454] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 0.
I20260924 11:48:06.041467 9016 leader_election.cc:302] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c3ce6361a0c481eaf81bc1b655b8a53, 90f2d46a63fa478b82cead56cfaa6259, cfde18f86de144a59dde6f2436d80440; no voters:
I20260924 11:48:06.041760 9372 raft_consensus.cc:2781] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:06.041903 9372 raft_consensus.cc:482] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:06.042011 9372 raft_consensus.cc:3037] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:06.044034 9372 raft_consensus.cc:504] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.044521 9372 leader_election.cc:288] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 election: Requested vote from peers 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009), cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459)
I20260924 11:48:06.045040 8934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b074f3288b44521b38f7a89413783b9" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53"
I20260924 11:48:06.045040 9222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b074f3288b44521b38f7a89413783b9" candidate_uuid: "90f2d46a63fa478b82cead56cfaa6259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde18f86de144a59dde6f2436d80440"
I20260924 11:48:06.045272 9222 raft_consensus.cc:3037] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:06.045274 8934 raft_consensus.cc:3037] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:06.048164 9222 raft_consensus.cc:2454] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 1.
I20260924 11:48:06.048158 8934 raft_consensus.cc:2454] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d46a63fa478b82cead56cfaa6259 in term 1.
I20260924 11:48:06.048553 9017 leader_election.cc:302] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f2d46a63fa478b82cead56cfaa6259, cfde18f86de144a59dde6f2436d80440; no voters:
I20260924 11:48:06.048826 9372 raft_consensus.cc:2781] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:06.049279 9372 raft_consensus.cc:686] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 LEADER]: Becoming Leader. State: Replica: 90f2d46a63fa478b82cead56cfaa6259, State: Running, Role: LEADER
I20260924 11:48:06.049557 9372 consensus_queue.cc:237] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } }
I20260924 11:48:06.050431 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.049936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:54951)
0924 11:48:06.050103 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.050247 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 11:48:06.052388 8608 catalog_manager.cc:5495] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "90f2d46a63fa478b82cead56cfaa6259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f2d46a63fa478b82cead56cfaa6259" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 42945 } health_report { overall_health: HEALTHY } } }
I20260924 11:48:06.083456 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.079097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.1.210.2:42945)
0924 11:48:06.079307 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:06.079311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:06.079328 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:48:06.079481 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:06.079488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:06.079563 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:06.079643 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:06.079959 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.080754 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.082076 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.082478 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.082586 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:06.082925 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:06.083015 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:06.083079 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:06.083202 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:48:06.083259 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:48:06.086452 9222 raft_consensus.cc:1261] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Refusing update from remote peer 90f2d46a63fa478b82cead56cfaa6259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:06.086452 8934 raft_consensus.cc:1261] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Refusing update from remote peer 90f2d46a63fa478b82cead56cfaa6259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:06.087090 9375 consensus_queue.cc:1035] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3ce6361a0c481eaf81bc1b655b8a53" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:06.087435 9372 consensus_queue.cc:1035] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfde18f86de144a59dde6f2436d80440" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:06.090201 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.089736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.3:41649)
0924 11:48:06.089922 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.090022 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 11:48:06.090593 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.090170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:42741)
0924 11:48:06.090311 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.090419 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 11:48:06.097959 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.097510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.3:40573)
0924 11:48:06.097708 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.097808 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 11:48:06.099135 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.098535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.1:49277)
0924 11:48:06.098784 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.098902 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 11:48:06.105974 8607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:06.106244 8607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:06.106909 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.106168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:39231)
0924 11:48:06.106666 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.106758 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387}
I20260924 11:48:06.108314 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.107981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.1.210.2:43405)
0924 11:48:06.108142 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.108204 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 11:48:06.112879 8607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebb5374cc6164b31acbed1081a4843e7: Sending DeleteTablet for 3 replicas of tablet 5b074f3288b44521b38f7a89413783b9
I20260924 11:48:06.113781 8914 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b074f3288b44521b38f7a89413783b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:06 UTC) from {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:40918
I20260924 11:48:06.114042 9202 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b074f3288b44521b38f7a89413783b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:06 UTC) from {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:33878
I20260924 11:48:06.114087 9061 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b074f3288b44521b38f7a89413783b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:06 UTC) from {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:51922
I20260924 11:48:06.114215 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b074f3288b44521b38f7a89413783b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:06 UTC) from {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:51922
I20260924 11:48:06.114539 9201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b074f3288b44521b38f7a89413783b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:06 UTC) from {username='oryx', principal='oryx/127.1.210.61@KRBTEST.COM'} at 127.0.0.1:33878
I20260924 11:48:06.118541 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8788
I20260924 11:48:06.124954 9389 tablet_replica.cc:330] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: stopping tablet replica
I20260924 11:48:06.125272 9389 raft_consensus.cc:2229] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:06.125784 9389 raft_consensus.cc:2258] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:06.126307 9387 tablet_replica.cc:330] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: stopping tablet replica
I20260924 11:48:06.126549 9387 raft_consensus.cc:2229] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:06.127100 9387 raft_consensus.cc:2258] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:06.127118 9389 ts_tablet_manager.cc:1905] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:06.128546 9387 ts_tablet_manager.cc:1905] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:06.135023 9387 ts_tablet_manager.cc:1918] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:06.135170 9387 log.cc:1196] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/5b074f3288b44521b38f7a89413783b9
I20260924 11:48:06.135617 9387 ts_tablet_manager.cc:1939] T 5b074f3288b44521b38f7a89413783b9 P 4c3ce6361a0c481eaf81bc1b655b8a53: Deleting consensus metadata
I20260924 11:48:06.136018 9389 ts_tablet_manager.cc:1918] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:06.136180 9389 log.cc:1196] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/5b074f3288b44521b38f7a89413783b9
I20260924 11:48:06.136802 9388 tablet_replica.cc:330] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: stopping tablet replica
I20260924 11:48:06.136817 8594 catalog_manager.cc:4841] TS 4c3ce6361a0c481eaf81bc1b655b8a53 (127.1.210.1:44009): tablet 5b074f3288b44521b38f7a89413783b9 (table test-table [id=786a997131eb4022b8680a83e8ccf707]) successfully deleted
I20260924 11:48:06.137023 9388 raft_consensus.cc:2229] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:06.137382 9388 raft_consensus.cc:2258] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:06.137982 9389 ts_tablet_manager.cc:1939] T 5b074f3288b44521b38f7a89413783b9 P 90f2d46a63fa478b82cead56cfaa6259: Deleting consensus metadata
I20260924 11:48:06.138376 9388 ts_tablet_manager.cc:1905] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:06.139006 8594 catalog_manager.cc:4841] TS 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2:42945): tablet 5b074f3288b44521b38f7a89413783b9 (table test-table [id=786a997131eb4022b8680a83e8ccf707]) successfully deleted
W20260924 11:48:06.139976 8594 catalog_manager.cc:4808] TS 90f2d46a63fa478b82cead56cfaa6259 (127.1.210.2:42945): delete failed for tablet 5b074f3288b44521b38f7a89413783b9 because the tablet was not found. No further retry: Not found: Tablet not found: 5b074f3288b44521b38f7a89413783b9
I20260924 11:48:06.144156 9388 ts_tablet_manager.cc:1918] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:06.144258 9388 log.cc:1196] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/5b074f3288b44521b38f7a89413783b9
I20260924 11:48:06.144675 9388 ts_tablet_manager.cc:1939] T 5b074f3288b44521b38f7a89413783b9 P cfde18f86de144a59dde6f2436d80440: Deleting consensus metadata
I20260924 11:48:06.145784 8595 catalog_manager.cc:4841] TS cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459): tablet 5b074f3288b44521b38f7a89413783b9 (table test-table [id=786a997131eb4022b8680a83e8ccf707]) successfully deleted
W20260924 11:48:06.146054 8595 catalog_manager.cc:4808] TS cfde18f86de144a59dde6f2436d80440 (127.1.210.3:46459): delete failed for tablet 5b074f3288b44521b38f7a89413783b9 because the tablet was not found. No further retry: Not found: Tablet not found: 5b074f3288b44521b38f7a89413783b9
I20260924 11:48:06.147822 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.119618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.1.210.1:44009)
0924 11:48:06.119854 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:06.119860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:06.130354 (+ 10494us) server_negotiation.cc:407] Connection header received
0924 11:48:06.130566 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:06.130572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:06.130665 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:06.130764 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:06.134874 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.136013 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.137886 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.138519 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.138660 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:06.141904 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:06.141917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:06.141931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:06.141960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:06.144837 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:06.145641 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:06.145650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:06.145653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:06.145796 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:06.146342 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:06.146347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:06.146350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:06.146698 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:06.146998 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:06.147285 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 11:48:06.147496 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:48:06.148447 8974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:06.241412 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8788
I20260924 11:48:06.258446 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8991
I20260924 11:48:06.271409 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.259196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.1.210.2:42945)
0924 11:48:06.259374 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:06.259382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:06.259444 (+ 62us) server_negotiation.cc:407] Connection header received
0924 11:48:06.259666 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:06.259672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:06.259747 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:06.259863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:06.261128 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.261841 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.263097 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.263521 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.263656 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:06.266137 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:06.266151 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:06.266167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:06.266198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:06.268706 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:06.269406 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:06.269416 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:06.269420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:06.269545 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:06.269909 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:06.269915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:06.269918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:06.270281 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:06.270569 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:06.270876 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 11:48:06.271091 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:48:06.272042 9121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:06.348261 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.347836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57290)
0924 11:48:06.348031 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.348121 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 11:48:06.349325 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.348950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43409 (local address 127.0.0.1:57302)
0924 11:48:06.349133 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:06.349212 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 11:48:06.393219 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8991
I20260924 11:48:06.417156 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 9139
I20260924 11:48:06.429354 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:06.417927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.1.210.3:46459)
0924 11:48:06.418133 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:06.418137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:06.418157 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:48:06.418279 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:06.418283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:06.418351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:06.418430 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:06.419647 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.420330 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.421563 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:06.421967 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:06.422058 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:06.424388 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:06.424400 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:06.424416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:06.424444 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:06.427085 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:06.427647 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:06.427653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:06.427656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:06.427768 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:06.428062 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:06.428065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:06.428067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:06.428379 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:06.428577 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:06.428824 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 11:48:06.428979 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:48:06.430003 9262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:06.522092 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 9139
I20260924 11:48:06.539076 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8471
I20260924 11:48:06.540067 8638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:06.659317 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8471
I20260924 11:48:06.680076 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 8570
I20260924 11:48:06.681277 8829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:06.793145 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 8570
2026-09-24T11:48:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9367 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:48:06 dist-test-slave-dd32 krb5kdc[9398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:48:06 dist-test-slave-dd32 krb5kdc[9398](info): set up 2 sockets
Sep 24 11:48:06 dist-test-slave-dd32 krb5kdc[9398](info): commencing operation
krb5kdc: starting...
W20260924 11:48:07.894070 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:48:08 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab.
Entry for principal oryx/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab.
I20260924 11:48:08.093008 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37493
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab
--principal=oryx/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:37493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:08.423852 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:08.424177 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:08.424265 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:08.431186 9411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:08.431277 9411 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:48:08.431316 9411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:08.431373 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:08.431416 9411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:08.431497 9411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:08.441025 9411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:37493
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.62.keytab
--principal=oryx/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:08.442282 9411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:08.443562 9411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:08.455191 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:08.455309 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:08.455293 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:08.456125 9411 server_base.cc:1034] running on GCE node
I20260924 11:48:08.456863 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:08.458240 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:08.459482 9411 hybrid_clock.cc:648] HybridClock initialized: now 1790250488459426 us; error 57 us; skew 500 ppm
I20260924 11:48:08.459803 9411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:08 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250488, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:08.462824 9411 init.cc:377] Logged in from keytab as oryx/127.1.210.62@KRBTEST.COM (short username oryx)
I20260924 11:48:08.464087 9411 webserver.cc:460] Webserver started at http://127.1.210.62:39651/ using document root <none> and password file <none>
I20260924 11:48:08.464754 9411 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:08.464877 9411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:08.465301 9411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:08.468101 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "403fa00ab15243ccaa845eedd1e376e8"
format_stamp: "Formatted at 2026-09-24 11:48:08 on dist-test-slave-dd32"
server_key: "dd7b7ddd32ce5a48af257fc96c4ca294"
server_key_iv: "9ddb6dd6bb7df72e0c1b737b928d1ad8"
server_key_version: "encryptionkey@0"
I20260924 11:48:08.468874 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "403fa00ab15243ccaa845eedd1e376e8"
format_stamp: "Formatted at 2026-09-24 11:48:08 on dist-test-slave-dd32"
server_key: "dd7b7ddd32ce5a48af257fc96c4ca294"
server_key_iv: "9ddb6dd6bb7df72e0c1b737b928d1ad8"
server_key_version: "encryptionkey@0"
I20260924 11:48:08.473883 9411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:48:08.477161 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:08.478390 9411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:48:08.478561 9411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "403fa00ab15243ccaa845eedd1e376e8"
format_stamp: "Formatted at 2026-09-24 11:48:08 on dist-test-slave-dd32"
server_key: "dd7b7ddd32ce5a48af257fc96c4ca294"
server_key_iv: "9ddb6dd6bb7df72e0c1b737b928d1ad8"
server_key_version: "encryptionkey@0"
I20260924 11:48:08.478765 9411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:08.493532 9411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:08.497220 9411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:08.497468 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:08.518390 9411 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:37493
I20260924 11:48:08.518411 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:37493 every 8 connection(s)
I20260924 11:48:08.519739 9411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:48:08.521569 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 9411
I20260924 11:48:08.521791 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:48:08.522615 1864 external_mini_cluster.cc:1428] Setting key f75157f718e47062850f55e3466688be
I20260924 11:48:08.524302 9488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:48:08 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:08.537699 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Bootstrap starting.
I20260924 11:48:08.541798 9488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:08.542976 9488 log.cc:824] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:08.546038 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:08.524606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.1.210.62:37493)
0924 11:48:08.525401 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:08.525413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:08.525463 (+ 50us) server_negotiation.cc:407] Connection header received
0924 11:48:08.526987 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:08.527006 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:08.527397 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:08.527750 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:08.529524 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:08.530864 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:08.532565 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:08.533355 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:08.533480 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:08.537844 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:08.537854 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:08.537871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:08.537914 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:08.541748 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:08.542398 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:08.542406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:08.542412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:08.542607 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:08.542952 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:08.542959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:08.542967 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:08.543634 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:08.543868 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:08.544387 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 11:48:08.544772 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":406,"threads_started":1}
I20260924 11:48:08.546142 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: No bootstrap required, opened a new log
I20260924 11:48:08.553714 9488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403fa00ab15243ccaa845eedd1e376e8" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37493 } }
I20260924 11:48:08.554004 9488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:08.554072 9488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403fa00ab15243ccaa845eedd1e376e8, State: Initialized, Role: FOLLOWER
I20260924 11:48:08.554572 9488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403fa00ab15243ccaa845eedd1e376e8" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37493 } }
I20260924 11:48:08.554780 9488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:48:08.554872 9488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:48:08.554984 9488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:08.556877 9488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403fa00ab15243ccaa845eedd1e376e8" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37493 } }
I20260924 11:48:08.557350 9488 leader_election.cc:302] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403fa00ab15243ccaa845eedd1e376e8; no voters:
I20260924 11:48:08.557827 9488 leader_election.cc:288] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:48:08.557997 9493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:08.558754 9493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [term 1 LEADER]: Becoming Leader. State: Replica: 403fa00ab15243ccaa845eedd1e376e8, State: Running, Role: LEADER
I20260924 11:48:08.559355 9493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403fa00ab15243ccaa845eedd1e376e8" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37493 } }
I20260924 11:48:08.559929 9488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:08.562358 9495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403fa00ab15243ccaa845eedd1e376e8. Latest consensus state: current_term: 1 leader_uuid: "403fa00ab15243ccaa845eedd1e376e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403fa00ab15243ccaa845eedd1e376e8" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37493 } } }
I20260924 11:48:08.562608 9494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "403fa00ab15243ccaa845eedd1e376e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403fa00ab15243ccaa845eedd1e376e8" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37493 } } }
I20260924 11:48:08.562965 9495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:08.562978 9494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:08.564455 9502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:08.570348 9502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:08.579025 9502 catalog_manager.cc:1347] Generated new cluster ID: 988c66b6c65446c18a11d1198c58b434
I20260924 11:48:08.579128 9502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:08.625566 9502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:48:08.626864 9502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:08.639103 9502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Generated new TSK 0
I20260924 11:48:08.639947 9502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab.
Entry for principal oryx/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab.
I20260924 11:48:08.704160 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37493
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:09.028776 9514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:09.029065 9514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:09.029259 9514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:09.036381 9514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:09.036497 9514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:09.036645 9514 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:48:09.036768 9514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:09.046777 9514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:09.047976 9514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:09.049347 9514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:09.062940 9522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:09.062915 9520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:09.063022 9519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:09.063460 9514 server_base.cc:1034] running on GCE node
I20260924 11:48:09.064153 9514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:09.065223 9514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:09.066434 9514 hybrid_clock.cc:648] HybridClock initialized: now 1790250489066372 us; error 46 us; skew 500 ppm
I20260924 11:48:09.066745 9514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:09 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250489, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:09.069944 9514 init.cc:377] Logged in from keytab as oryx/127.1.210.1@KRBTEST.COM (short username oryx)
I20260924 11:48:09.071420 9514 webserver.cc:460] Webserver started at http://127.1.210.1:45857/ using document root <none> and password file <none>
I20260924 11:48:09.072044 9514 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:09.072126 9514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:09.072382 9514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:09.075295 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "73442790fe5145c58a43cd75d600871a"
format_stamp: "Formatted at 2026-09-24 11:48:09 on dist-test-slave-dd32"
server_key: "6f4f2855a121c44057722471b52119a4"
server_key_iv: "2bab071581349f9788e2db962d17ff07"
server_key_version: "encryptionkey@0"
I20260924 11:48:09.076030 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "73442790fe5145c58a43cd75d600871a"
format_stamp: "Formatted at 2026-09-24 11:48:09 on dist-test-slave-dd32"
server_key: "6f4f2855a121c44057722471b52119a4"
server_key_iv: "2bab071581349f9788e2db962d17ff07"
server_key_version: "encryptionkey@0"
I20260924 11:48:09.080900 9514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260924 11:48:09.084386 9529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:09.085763 9514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:09.085932 9514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "73442790fe5145c58a43cd75d600871a"
format_stamp: "Formatted at 2026-09-24 11:48:09 on dist-test-slave-dd32"
server_key: "6f4f2855a121c44057722471b52119a4"
server_key_iv: "2bab071581349f9788e2db962d17ff07"
server_key_version: "encryptionkey@0"
I20260924 11:48:09.086136 9514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:09.124279 9514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:09.128343 9514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:09.128618 9514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:09.131208 9514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:09.131292 9514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:09.131435 9514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:09.131505 9514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:48:09 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250489, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:09.156837 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.131983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:36225 (local address 127.1.210.62:37493)
0924 11:48:09.132430 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:09.132437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:09.133807 (+ 1370us) server_negotiation.cc:407] Connection header received
0924 11:48:09.137313 (+ 3506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:09.137320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:09.137419 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:09.137541 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:09.139418 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.140448 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.142879 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.143501 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.143654 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.148974 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:09.148979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:09.148982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:09.149012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:09.152479 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.153645 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.153651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.153655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.153779 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.154396 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.154401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.154405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.154840 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:09.155014 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:09.156035 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0924 11:48:09.156322 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260924 11:48:09.158413 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.132695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.1.210.1:36225)
0924 11:48:09.133610 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:09.133661 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:09.136878 (+ 3217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:09.137809 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:09.137823 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:09.138537 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:09.139179 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.139204 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.140608 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.140616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.142734 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.142746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.143879 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.143889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.144335 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.145371 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:09.145409 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:09.148728 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:09.152634 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.152644 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.152665 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.153467 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.153906 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.153911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.153914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.154192 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.155008 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:09.155020 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:09.155569 (+ 549us) client_negotiation.cc:769] Sending connection context
0924 11:48:09.155900 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 11:48:09.156580 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":403,"threads_started":1}
I20260924 11:48:09.179112 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.164495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.1.210.1:59003)
0924 11:48:09.165251 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:09.165271 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:09.165485 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:09.165885 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:09.165890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:09.166348 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:09.166641 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.166649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.167893 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.167901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.169745 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.169769 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.170842 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.170851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.171085 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.171796 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:09.171834 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:09.173160 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:09.176013 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.176021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.176028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.176664 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.177243 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.177248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.177251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.177436 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.178102 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:09.178106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:09.178349 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 11:48:09.178527 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 11:48:09.178802 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":364,"threads_started":1}
I20260924 11:48:09.179306 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.164499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:59003 (local address 127.1.210.62:37493)
0924 11:48:09.164712 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:09.164720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:09.165354 (+ 634us) server_negotiation.cc:407] Connection header received
0924 11:48:09.165681 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:09.165686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:09.165755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:09.165855 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:09.166779 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.167730 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.169905 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.170619 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.170757 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.173346 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:09.173351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:09.173355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:09.173383 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:09.175890 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.176830 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.176836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.176843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.177033 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.177627 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.177635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.177641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.177979 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:09.178130 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:09.178664 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 11:48:09.178906 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:48:09.180322 9514 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:46403
I20260924 11:48:09.180409 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:46403 every 8 connection(s)
I20260924 11:48:09.181715 9514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:09.184085 9650 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37493
I20260924 11:48:09.184310 9650 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:09.184899 9650 heartbeater.cc:511] Master 127.1.210.62:37493 requested a full tablet report, sending...
I20260924 11:48:09.186225 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 9514
I20260924 11:48:09.186429 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:48:09.186882 9443 ts_manager.cc:194] Registered new tserver with Master: 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403)
I20260924 11:48:09.187093 1864 external_mini_cluster.cc:1428] Setting key 4565027f8b0bee6a7d580e5b9f0b338e
I20260924 11:48:09.188743 9443 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:36225
WARNING: no policy specified for oryx/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab.
Entry for principal oryx/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab.
I20260924 11:48:09.231909 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37493
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:09.546522 9654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:09.546813 9654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:09.546980 9654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:09.554116 9654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:09.554278 9654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:09.554425 9654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:48:09.554561 9654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:09.564437 9654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:09.565812 9654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:09.567238 9654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:09.580595 9659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:09.580560 9662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:09.580629 9660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:09.581421 9654 server_base.cc:1034] running on GCE node
I20260924 11:48:09.582084 9654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:09.583048 9654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:09.584275 9654 hybrid_clock.cc:648] HybridClock initialized: now 1790250489584211 us; error 43 us; skew 500 ppm
I20260924 11:48:09.584589 9654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:09 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250489, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:09.587668 9654 init.cc:377] Logged in from keytab as oryx/127.1.210.2@KRBTEST.COM (short username oryx)
I20260924 11:48:09.589048 9654 webserver.cc:460] Webserver started at http://127.1.210.2:33637/ using document root <none> and password file <none>
I20260924 11:48:09.589788 9654 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:09.589908 9654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:09.590240 9654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:09.592976 9654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "a60fe945aa114939bc98346691650908"
format_stamp: "Formatted at 2026-09-24 11:48:09 on dist-test-slave-dd32"
server_key: "188c6f6c05f947e5a4b85e9a2ed660c0"
server_key_iv: "c2fd67ffd2858f7f7f99a92b165e4270"
server_key_version: "encryptionkey@0"
I20260924 11:48:09.593868 9654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "a60fe945aa114939bc98346691650908"
format_stamp: "Formatted at 2026-09-24 11:48:09 on dist-test-slave-dd32"
server_key: "188c6f6c05f947e5a4b85e9a2ed660c0"
server_key_iv: "c2fd67ffd2858f7f7f99a92b165e4270"
server_key_version: "encryptionkey@0"
I20260924 11:48:09.598706 9654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:48:09.601935 9669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:09.603155 9654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:48:09.603338 9654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "a60fe945aa114939bc98346691650908"
format_stamp: "Formatted at 2026-09-24 11:48:09 on dist-test-slave-dd32"
server_key: "188c6f6c05f947e5a4b85e9a2ed660c0"
server_key_iv: "c2fd67ffd2858f7f7f99a92b165e4270"
server_key_version: "encryptionkey@0"
I20260924 11:48:09.603542 9654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:09.627060 9654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:09.631393 9654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:09.631661 9654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:09.634310 9654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:09.634454 9654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:09.634654 9654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:09.634745 9654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:48:09 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250489, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:09.655781 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.634787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:50031 (local address 127.1.210.62:37493)
0924 11:48:09.634954 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:09.634959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:09.636414 (+ 1455us) server_negotiation.cc:407] Connection header received
0924 11:48:09.638495 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:09.638501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:09.638620 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:09.638773 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:09.640688 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.641974 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.644721 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.645422 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.645569 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.649359 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:09.649364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:09.649367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:09.649396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:09.652312 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.653227 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.653234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.653241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.653400 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.653954 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.653962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.653965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.654366 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:09.654529 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:09.655201 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 11:48:09.655420 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:48:09.656756 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.635268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.1.210.2:50031)
0924 11:48:09.636228 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:09.636281 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:09.638226 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:09.639046 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:09.639062 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:09.639580 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:09.640469 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.640501 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.642182 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.642189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.644530 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.644561 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.645698 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.645710 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.645903 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.646574 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:09.646598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:09.649155 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:09.652449 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.652459 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.652478 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.653074 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.653559 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.653570 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.653573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.653830 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.654507 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:09.654515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:09.654847 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 11:48:09.655096 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 11:48:09.655583 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":441,"threads_started":1}
I20260924 11:48:09.678717 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.663296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:56109 (local address 127.1.210.62:37493)
0924 11:48:09.663468 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:09.663475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:09.664182 (+ 707us) server_negotiation.cc:407] Connection header received
0924 11:48:09.664361 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:09.664367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:09.664435 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:09.664531 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:09.665657 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.666942 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.668890 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.669520 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.669680 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.671750 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:09.671760 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:09.671767 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:09.671816 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:09.675767 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.676656 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.676663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.676667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.676840 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:09.677300 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:09.677306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:09.677309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:09.677615 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:09.677787 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:09.678182 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 11:48:09.678383 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:48:09.678872 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:09.663407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.1.210.2:56109)
0924 11:48:09.664068 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:09.664086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:09.664235 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:09.664551 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:09.664556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:09.665008 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:09.665479 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.665498 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.667093 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.667100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.668749 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:09.668761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:09.669955 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:09.669960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:09.670215 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:09.670868 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:09.670885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:09.671528 (+ 643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:09.675968 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.675976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.675982 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.676529 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.676975 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:09.676986 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:09.676995 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:09.677177 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:09.677740 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:09.677745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:09.677934 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 11:48:09.678090 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 11:48:09.678397 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":387,"threads_started":1}
I20260924 11:48:09.688454 9654 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:35375
I20260924 11:48:09.688476 9788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:35375 every 8 connection(s)
I20260924 11:48:09.689775 9654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:09.690788 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 9654
I20260924 11:48:09.690969 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:48:09.691478 1864 external_mini_cluster.cc:1428] Setting key 32a645462fd36dcf8e9274b004fc4aea
I20260924 11:48:09.691977 9789 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37493
I20260924 11:48:09.692270 9789 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:09.694072 9789 heartbeater.cc:511] Master 127.1.210.62:37493 requested a full tablet report, sending...
I20260924 11:48:09.695474 9443 ts_manager.cc:194] Registered new tserver with Master: a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:09.696271 9443 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:50031
WARNING: no policy specified for oryx/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab.
Entry for principal oryx/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab.
I20260924 11:48:09.736243 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37493
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:10.048331 9793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:10.048593 9793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:10.048723 9793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:10.055933 9793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:10.056031 9793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:10.056206 9793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:48:10.056325 9793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:10.066395 9793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:10.067626 9793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:10.068925 9793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:10.081764 9801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:10.081832 9798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:10.081794 9799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:10.082577 9793 server_base.cc:1034] running on GCE node
I20260924 11:48:10.083122 9793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:10.084019 9793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:10.085170 9793 hybrid_clock.cc:648] HybridClock initialized: now 1790250490085121 us; error 26 us; skew 500 ppm
I20260924 11:48:10.085481 9793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250490, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:10.088325 9793 init.cc:377] Logged in from keytab as oryx/127.1.210.3@KRBTEST.COM (short username oryx)
I20260924 11:48:10.089773 9793 webserver.cc:460] Webserver started at http://127.1.210.3:36573/ using document root <none> and password file <none>
I20260924 11:48:10.090489 9793 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:10.090590 9793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:10.090909 9793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:10.093663 9793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "c777ec3504384b6db187df4487ada918"
format_stamp: "Formatted at 2026-09-24 11:48:10 on dist-test-slave-dd32"
server_key: "135878f6d6c54feea0849299d03a655f"
server_key_iv: "ee5a326f8bb09bae4e2baaeba7d1fc3b"
server_key_version: "encryptionkey@0"
I20260924 11:48:10.094389 9793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "c777ec3504384b6db187df4487ada918"
format_stamp: "Formatted at 2026-09-24 11:48:10 on dist-test-slave-dd32"
server_key: "135878f6d6c54feea0849299d03a655f"
server_key_iv: "ee5a326f8bb09bae4e2baaeba7d1fc3b"
server_key_version: "encryptionkey@0"
I20260924 11:48:10.099062 9793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:10.102317 9808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:10.103520 9793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:10.103700 9793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "c777ec3504384b6db187df4487ada918"
format_stamp: "Formatted at 2026-09-24 11:48:10 on dist-test-slave-dd32"
server_key: "135878f6d6c54feea0849299d03a655f"
server_key_iv: "ee5a326f8bb09bae4e2baaeba7d1fc3b"
server_key_version: "encryptionkey@0"
I20260924 11:48:10.103917 9793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:10.124446 9793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:10.128118 9793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:10.128407 9793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:10.130903 9793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:10.131016 9793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:10.131194 9793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:10.131273 9793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250490, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:10.156270 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.131661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:55513 (local address 127.1.210.62:37493)
0924 11:48:10.131857 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.131862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.133275 (+ 1413us) server_negotiation.cc:407] Connection header received
0924 11:48:10.135285 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.135291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.135378 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.135492 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:10.137636 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.138944 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.141484 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.142103 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.142238 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.147847 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:10.147854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:10.147859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:10.147907 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:10.151551 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.152600 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.152609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.152614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.152815 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.153570 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.153579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.153584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.154012 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:10.154170 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.155484 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.155807 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:48:10.158213 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.132235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.1.210.3:55513)
0924 11:48:10.133093 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.133143 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.135012 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.135756 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.135769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.136576 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:10.137405 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.137449 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.139164 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.139170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.141336 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.141351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.142387 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.142401 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.142829 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.143696 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:10.143746 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:10.147537 (+ 3791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:10.151707 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:10.151724 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:10.151752 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:10.152440 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:10.152998 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:10.153006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:10.153013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:10.153396 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:10.154264 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:10.154276 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:10.154910 (+ 634us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.155316 (+ 406us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.156094 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":366,"threads_started":1}
I20260924 11:48:10.181154 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.165471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.1.210.3:39669)
0924 11:48:10.166178 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.166195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.166348 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.166659 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.166664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.167098 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:10.167369 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.167377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.168344 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.168351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.169782 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.169799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.170858 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.170867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.171108 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.171875 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:10.171904 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:10.173028 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:10.177640 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:10.177652 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:10.177658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:10.178226 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:10.178695 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:10.178700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:10.178703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:10.178867 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:10.179582 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:10.179586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:10.179813 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.179995 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.180477 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":407,"threads_started":1}
I20260924 11:48:10.182351 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.165353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:39669 (local address 127.1.210.62:37493)
0924 11:48:10.165529 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.165536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.166279 (+ 743us) server_negotiation.cc:407] Connection header received
0924 11:48:10.166462 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.166467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.166537 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.166638 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:10.167507 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.168210 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.169980 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.170652 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.170800 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.173251 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:10.173261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:10.173269 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:10.173320 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:10.177495 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.178390 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.178395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.178398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.178553 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.179005 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.179010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.179013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.179367 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:10.179516 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.181510 (+ 1994us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.181843 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:48:10.186146 9793 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:34987
I20260924 11:48:10.186239 9927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:34987 every 8 connection(s)
I20260924 11:48:10.187362 9793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:10.189476 9928 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37493
I20260924 11:48:10.189710 9928 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:10.190364 9928 heartbeater.cc:511] Master 127.1.210.62:37493 requested a full tablet report, sending...
I20260924 11:48:10.191589 9443 ts_manager.cc:194] Registered new tserver with Master: c777ec3504384b6db187df4487ada918 (127.1.210.3:34987)
I20260924 11:48:10.192173 9650 heartbeater.cc:503] Master 127.1.210.62:37493 was elected leader, sending a full tablet report...
I20260924 11:48:10.192595 9443 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:55513
I20260924 11:48:10.195789 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 9793
I20260924 11:48:10.195910 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:48:10.196285 1864 external_mini_cluster.cc:1428] Setting key 397252dcfcef65c48aaeb8b3fa104f75
I20260924 11:48:10.198835 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250490, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:48:10.246124 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:10.235950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.1.210.62:37493)
0924 11:48:10.236160 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.236167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.236350 (+ 183us) server_negotiation.cc:407] Connection header received
0924 11:48:10.236558 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.236563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.236654 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.236769 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:10.238159 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.238893 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.240100 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.240545 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.240674 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.245766 (+ 5092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58640: BlockingRecv error: recv got EOF from 127.0.0.1:58640 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250490, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:10.261392 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.248815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.1.210.62:37493)
0924 11:48:10.248954 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.248958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.249323 (+ 365us) server_negotiation.cc:407] Connection header received
0924 11:48:10.249437 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.249440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.249504 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.249580 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:10.250552 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.251221 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.252336 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.252800 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.252943 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.256330 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:10.256335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:10.256338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:10.256366 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:10.258744 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.259401 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.259407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.259411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.259633 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.260049 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.260071 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.260076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.260345 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:10.260606 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.260906 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.261049 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:48:10.264894 9442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:10.271834 9442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:10.293526 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.283496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:34987 (local address 127.0.0.1:39762)
0924 11:48:10.284482 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.284499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.284596 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.285722 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.285729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.285764 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:10.286149 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.286166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.288463 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.288471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.290690 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.290706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.292864 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.292868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.292982 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.293011 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.293155 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.293293 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":2777,"threads_started":1}
I20260924 11:48:10.294265 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.283748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.1.210.3:34987)
0924 11:48:10.285240 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.285246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.285271 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:48:10.285344 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.285349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.285515 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.285835 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:10.286924 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.288326 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.291159 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.292512 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.292591 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.292759 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.293930 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.294042 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":1140,"threads_started":1}
I20260924 11:48:10.294592 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.282825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:35375 (local address 127.0.0.1:45058)
0924 11:48:10.283528 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.283564 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.283742 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.287449 (+ 3707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.287453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.287478 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:10.287892 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.287905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.289863 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.289869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.292117 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.292132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.293854 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.293860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.294028 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.294072 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.294135 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.294254 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":356,"threads_started":1}
I20260924 11:48:10.295269 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.284217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.1.210.1:46403)
0924 11:48:10.287206 (+ 2989us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.287214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.287239 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:48:10.287328 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.287337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.287556 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.287743 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:10.288515 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.289968 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.292157 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.293615 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.293753 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.293952 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.294839 (+ 887us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.294989 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2817,"thread_start_us":2694,"threads_started":1}
I20260924 11:48:10.295506 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.283029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.1.210.2:35375)
0924 11:48:10.286541 (+ 3512us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.286553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.286578 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:48:10.286668 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.286679 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.287253 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.288071 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:10.288270 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.289753 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.292285 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.293633 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.294837 (+ 1204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.295010 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.295074 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.295218 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3365,"thread_start_us":3448,"threads_started":1}
I20260924 11:48:10.295720 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.283927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46403 (local address 127.0.0.1:59984)
0924 11:48:10.286513 (+ 2586us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.286530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.286633 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.287942 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.287948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.287975 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:10.288327 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.288340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.290098 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.290106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.292013 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.292024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.294076 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.294081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.294245 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.294271 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.294349 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.294467 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2484,"thread_start_us":283,"threads_started":1}
I20260924 11:48:10.298089 9858 tablet_service.cc:1467] Processing CreateTablet for tablet e590cb743a544614b8d9e0ad7458118a (DEFAULT_TABLE table=test-table [id=68465d56496546599233624033514dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:10.298506 9719 tablet_service.cc:1467] Processing CreateTablet for tablet e590cb743a544614b8d9e0ad7458118a (DEFAULT_TABLE table=test-table [id=68465d56496546599233624033514dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:10.298955 9580 tablet_service.cc:1467] Processing CreateTablet for tablet e590cb743a544614b8d9e0ad7458118a (DEFAULT_TABLE table=test-table [id=68465d56496546599233624033514dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:10.299602 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e590cb743a544614b8d9e0ad7458118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.299650 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e590cb743a544614b8d9e0ad7458118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.300144 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e590cb743a544614b8d9e0ad7458118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.310842 9952 tablet_bootstrap.cc:492] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Bootstrap starting.
I20260924 11:48:10.313851 9953 tablet_bootstrap.cc:492] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Bootstrap starting.
I20260924 11:48:10.314626 9954 tablet_bootstrap.cc:492] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Bootstrap starting.
I20260924 11:48:10.314908 9952 tablet_bootstrap.cc:654] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.316423 9952 log.cc:824] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:10.318367 9953 tablet_bootstrap.cc:654] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.318956 9952 tablet_bootstrap.cc:492] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: No bootstrap required, opened a new log
I20260924 11:48:10.319227 9952 ts_tablet_manager.cc:1397] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:48:10.319566 9953 log.cc:824] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:10.320272 9954 tablet_bootstrap.cc:654] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.321359 9954 log.cc:824] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:10.322435 9953 tablet_bootstrap.cc:492] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: No bootstrap required, opened a new log
I20260924 11:48:10.322707 9953 ts_tablet_manager.cc:1397] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:48:10.324131 9954 tablet_bootstrap.cc:492] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: No bootstrap required, opened a new log
I20260924 11:48:10.324370 9954 ts_tablet_manager.cc:1397] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 11:48:10.327329 9952 raft_consensus.cc:348] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.327670 9952 raft_consensus.cc:374] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.327772 9952 raft_consensus.cc:729] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fe945aa114939bc98346691650908, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.328361 9952 consensus_queue.cc:260] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.329404 9789 heartbeater.cc:503] Master 127.1.210.62:37493 was elected leader, sending a full tablet report...
I20260924 11:48:10.330062 9952 ts_tablet_manager.cc:1428] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 11:48:10.333024 9953 raft_consensus.cc:348] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.333500 9953 raft_consensus.cc:374] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.333604 9953 raft_consensus.cc:729] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73442790fe5145c58a43cd75d600871a, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.334235 9953 consensus_queue.cc:260] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.334774 9954 raft_consensus.cc:348] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.335965 9953 ts_tablet_manager.cc:1428] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 11:48:10.336191 9954 raft_consensus.cc:374] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.336316 9954 raft_consensus.cc:729] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c777ec3504384b6db187df4487ada918, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.336886 9954 consensus_queue.cc:260] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.338035 9928 heartbeater.cc:503] Master 127.1.210.62:37493 was elected leader, sending a full tablet report...
I20260924 11:48:10.338603 9954 ts_tablet_manager.cc:1428] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 11:48:10.375205 9960 raft_consensus.cc:482] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:10.375465 9960 raft_consensus.cc:504] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.377220 9960 leader_election.cc:288] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a60fe945aa114939bc98346691650908 (127.1.210.2:35375), 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403)
I20260924 11:48:10.383272 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.377552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:35375 (local address 127.1.210.3:35321)
0924 11:48:10.377762 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.377782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.377963 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.378393 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.378399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.378436 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:10.378675 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.378688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.379833 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.379840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.381186 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.381197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.382492 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.382497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.382726 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.382767 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.382863 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.382995 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260924 11:48:10.383320 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.377529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:35321 (local address 127.1.210.2:35375)
0924 11:48:10.377753 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.377759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.377868 (+ 109us) server_negotiation.cc:407] Connection header received
0924 11:48:10.378125 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.378133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.378224 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.378336 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:10.378841 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.379699 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.381337 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.382291 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.382416 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.382514 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.382902 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.383025 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:48:10.384538 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.378083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:44535 (local address 127.1.210.1:46403)
0924 11:48:10.378274 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.378278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.378933 (+ 655us) server_negotiation.cc:407] Connection header received
0924 11:48:10.379078 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.379083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.379154 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.379253 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:10.379741 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.380505 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.382435 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.383605 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.383730 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.383810 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.384295 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.384384 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:48:10.384409 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590cb743a544614b8d9e0ad7458118a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908" is_pre_election: true
I20260924 11:48:10.384702 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.377975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46403 (local address 127.1.210.3:44535)
0924 11:48:10.378831 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.378851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.378977 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.379275 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.379279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.379300 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:10.379591 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.379599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.380636 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.380642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.382292 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.382302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.383952 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.383958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.384130 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.384161 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.384269 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.384396 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":477,"threads_started":1}
I20260924 11:48:10.384979 9744 raft_consensus.cc:2454] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:10.385660 9811 leader_election.cc:302] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60fe945aa114939bc98346691650908, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:10.385743 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590cb743a544614b8d9e0ad7458118a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a" is_pre_election: true
I20260924 11:48:10.386066 9960 raft_consensus.cc:2781] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:10.386235 9960 raft_consensus.cc:482] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:10.386224 9600 raft_consensus.cc:2454] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:10.386312 9960 raft_consensus.cc:3037] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.388397 9960 raft_consensus.cc:504] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.388931 9960 leader_election.cc:288] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Requested vote from peers a60fe945aa114939bc98346691650908 (127.1.210.2:35375), 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403)
I20260924 11:48:10.389364 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590cb743a544614b8d9e0ad7458118a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908"
I20260924 11:48:10.389436 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590cb743a544614b8d9e0ad7458118a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a"
I20260924 11:48:10.389581 9744 raft_consensus.cc:3037] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.389638 9600 raft_consensus.cc:3037] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.391836 9600 raft_consensus.cc:2454] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:10.391839 9744 raft_consensus.cc:2454] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:10.392191 9811 leader_election.cc:302] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73442790fe5145c58a43cd75d600871a, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:10.392468 9960 raft_consensus.cc:2781] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:10.392902 9960 raft_consensus.cc:686] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Becoming Leader. State: Replica: c777ec3504384b6db187df4487ada918, State: Running, Role: LEADER
I20260924 11:48:10.393364 9960 consensus_queue.cc:237] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.397439 9442 catalog_manager.cc:5495] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 reported cstate change: term changed from 0 to 1, leader changed from <none> to c777ec3504384b6db187df4487ada918 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } health_report { overall_health: HEALTHY } } }
I20260924 11:48:10.417790 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.412866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.1.210.3:34987)
0924 11:48:10.413113 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.413120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.413221 (+ 101us) server_negotiation.cc:407] Connection header received
0924 11:48:10.413387 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.413390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.413468 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.413584 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:10.413881 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.414554 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.415752 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.416132 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.416232 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.416564 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:10.417109 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:10.417239 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.417446 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.417505 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:48:10.425130 9600 raft_consensus.cc:1261] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:10.425194 9744 raft_consensus.cc:1261] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:10.426017 9962 consensus_queue.cc:1035] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:10.426373 9960 consensus_queue.cc:1035] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:48:10.435082 9651 tablet.cc:2369] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:48:10.439159 9966 mvcc.cc:205] Tried to move back new op lower bound from 7332866008772509696 to 7332866008654622720. Current Snapshot: MvccSnapshot[applied={T|T < 7332866008772509696}]
I20260924 11:48:10.451417 9442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:10.451651 9442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:10.455065 9442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Sending DeleteTablet for 3 replicas of tablet e590cb743a544614b8d9e0ad7458118a
I20260924 11:48:10.456115 9719 tablet_service.cc:1514] Processing DeleteTablet for tablet e590cb743a544614b8d9e0ad7458118a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:10 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:45058
I20260924 11:48:10.456212 9580 tablet_service.cc:1514] Processing DeleteTablet for tablet e590cb743a544614b8d9e0ad7458118a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:10 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:59984
I20260924 11:48:10.456460 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet e590cb743a544614b8d9e0ad7458118a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:10 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:39762
I20260924 11:48:10.457643 9442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:10.458400 9442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:10.467221 9976 tablet_replica.cc:330] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: stopping tablet replica
I20260924 11:48:10.467350 9977 tablet_replica.cc:330] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: stopping tablet replica
I20260924 11:48:10.467590 9975 tablet_replica.cc:330] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: stopping tablet replica
I20260924 11:48:10.467828 9977 raft_consensus.cc:2229] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:10.468120 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 62337ec77c5a41dfa93060c012367f63 (DEFAULT_TABLE table=test-table [id=8b4def6e8cee4fc48ca5ab87f8e03fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:10.468482 9975 raft_consensus.cc:2229] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:10.468321 9579 tablet_service.cc:1467] Processing CreateTablet for tablet 62337ec77c5a41dfa93060c012367f63 (DEFAULT_TABLE table=test-table [id=8b4def6e8cee4fc48ca5ab87f8e03fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:10.468791 9976 raft_consensus.cc:2229] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:10.468595 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62337ec77c5a41dfa93060c012367f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.469132 9579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62337ec77c5a41dfa93060c012367f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.469941 9977 raft_consensus.cc:2258] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:10.469959 9976 raft_consensus.cc:2258] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:10.470121 9975 raft_consensus.cc:2258] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:10.473416 9977 ts_tablet_manager.cc:1905] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:10.475059 9975 ts_tablet_manager.cc:1905] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:10.475447 9952 tablet_bootstrap.cc:492] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Bootstrap starting.
I20260924 11:48:10.476943 9953 tablet_bootstrap.cc:492] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Bootstrap starting.
I20260924 11:48:10.477182 9976 ts_tablet_manager.cc:1905] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:10.477605 9952 tablet_bootstrap.cc:654] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.478545 9953 tablet_bootstrap.cc:654] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.478811 9858 tablet_service.cc:1467] Processing CreateTablet for tablet 62337ec77c5a41dfa93060c012367f63 (DEFAULT_TABLE table=test-table [id=8b4def6e8cee4fc48ca5ab87f8e03fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:10.479169 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62337ec77c5a41dfa93060c012367f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.480542 9953 tablet_bootstrap.cc:492] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: No bootstrap required, opened a new log
I20260924 11:48:10.480691 9953 ts_tablet_manager.cc:1397] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:48:10.481622 9952 tablet_bootstrap.cc:492] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: No bootstrap required, opened a new log
I20260924 11:48:10.481559 9953 raft_consensus.cc:348] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.481781 9952 ts_tablet_manager.cc:1397] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260924 11:48:10.481798 9953 raft_consensus.cc:374] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.481887 9953 raft_consensus.cc:729] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73442790fe5145c58a43cd75d600871a, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.482275 9953 consensus_queue.cc:260] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.482630 9952 raft_consensus.cc:348] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.482864 9952 raft_consensus.cc:374] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.482923 9953 ts_tablet_manager.cc:1428] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 11:48:10.482977 9952 raft_consensus.cc:729] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fe945aa114939bc98346691650908, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.483284 9952 consensus_queue.cc:260] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.486112 9976 ts_tablet_manager.cc:1918] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:10.486312 9976 log.cc:1196] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/e590cb743a544614b8d9e0ad7458118a
I20260924 11:48:10.487010 9976 ts_tablet_manager.cc:1939] T e590cb743a544614b8d9e0ad7458118a P 73442790fe5145c58a43cd75d600871a: Deleting consensus metadata
I20260924 11:48:10.488679 9975 ts_tablet_manager.cc:1918] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:10.488772 9429 catalog_manager.cc:4841] TS 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403): tablet e590cb743a544614b8d9e0ad7458118a (table test-table [id=68465d56496546599233624033514dab]) successfully deleted
I20260924 11:48:10.488950 9975 log.cc:1196] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/e590cb743a544614b8d9e0ad7458118a
I20260924 11:48:10.489549 9975 ts_tablet_manager.cc:1939] T e590cb743a544614b8d9e0ad7458118a P a60fe945aa114939bc98346691650908: Deleting consensus metadata
I20260924 11:48:10.490448 9954 tablet_bootstrap.cc:492] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Bootstrap starting.
I20260924 11:48:10.491259 9429 catalog_manager.cc:4841] TS a60fe945aa114939bc98346691650908 (127.1.210.2:35375): tablet e590cb743a544614b8d9e0ad7458118a (table test-table [id=68465d56496546599233624033514dab]) successfully deleted
I20260924 11:48:10.492172 9954 tablet_bootstrap.cc:654] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.493006 9952 ts_tablet_manager.cc:1428] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20260924 11:48:10.495332 9954 tablet_bootstrap.cc:492] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: No bootstrap required, opened a new log
I20260924 11:48:10.495468 9954 ts_tablet_manager.cc:1397] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 11:48:10.496331 9954 raft_consensus.cc:348] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.496578 9954 raft_consensus.cc:374] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.496680 9954 raft_consensus.cc:729] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c777ec3504384b6db187df4487ada918, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.496953 9954 consensus_queue.cc:260] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.497751 9954 ts_tablet_manager.cc:1428] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 11:48:10.499447 9977 ts_tablet_manager.cc:1918] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:10.499632 9977 log.cc:1196] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/e590cb743a544614b8d9e0ad7458118a
I20260924 11:48:10.500197 9977 ts_tablet_manager.cc:1939] T e590cb743a544614b8d9e0ad7458118a P c777ec3504384b6db187df4487ada918: Deleting consensus metadata
I20260924 11:48:10.501780 9428 catalog_manager.cc:4841] TS c777ec3504384b6db187df4487ada918 (127.1.210.3:34987): tablet e590cb743a544614b8d9e0ad7458118a (table test-table [id=68465d56496546599233624033514dab]) successfully deleted
I20260924 11:48:10.593482 9971 raft_consensus.cc:482] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:10.593719 9971 raft_consensus.cc:504] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.594527 9971 leader_election.cc:288] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:10.595031 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62337ec77c5a41dfa93060c012367f63" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a" is_pre_election: true
I20260924 11:48:10.595165 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62337ec77c5a41dfa93060c012367f63" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908" is_pre_election: true
I20260924 11:48:10.595284 9600 raft_consensus.cc:2454] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:10.595424 9744 raft_consensus.cc:2454] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:10.595755 9811 leader_election.cc:302] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73442790fe5145c58a43cd75d600871a, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:10.596030 9971 raft_consensus.cc:2781] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:10.596156 9971 raft_consensus.cc:482] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:10.596277 9971 raft_consensus.cc:3037] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.598481 9971 raft_consensus.cc:504] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.599035 9971 leader_election.cc:288] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Requested vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:10.599406 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62337ec77c5a41dfa93060c012367f63" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a"
I20260924 11:48:10.599603 9600 raft_consensus.cc:3037] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.599548 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62337ec77c5a41dfa93060c012367f63" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908"
I20260924 11:48:10.599731 9744 raft_consensus.cc:3037] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.601711 9600 raft_consensus.cc:2454] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:10.602072 9811 leader_election.cc:302] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73442790fe5145c58a43cd75d600871a, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:10.602347 9971 raft_consensus.cc:2781] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:10.602510 9971 raft_consensus.cc:686] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Becoming Leader. State: Replica: c777ec3504384b6db187df4487ada918, State: Running, Role: LEADER
I20260924 11:48:10.602701 9744 raft_consensus.cc:2454] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:10.602782 9971 consensus_queue.cc:237] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.605589 9443 catalog_manager.cc:5495] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 reported cstate change: term changed from 0 to 1, leader changed from <none> to c777ec3504384b6db187df4487ada918 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } health_report { overall_health: HEALTHY } } }
Sep 24 11:48:10 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250488, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.62@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:10.666888 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.651436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37493 (local address 127.0.0.1:58650)
0924 11:48:10.652153 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.652174 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.652331 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.652643 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.652650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.653615 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:10.653956 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.653968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.654966 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.654971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.656539 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.656552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.657259 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.657265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.657426 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.658092 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:10.658114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:10.661140 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:10.664634 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:10.664642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:10.664648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:10.665042 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:10.665408 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:10.665411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:10.665413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:10.665540 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:10.666036 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:10.666041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:10.666171 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.666303 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.666466 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":418,"threads_started":1}
I20260924 11:48:10.667114 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.651441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.1.210.62:37493)
0924 11:48:10.651798 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.651807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.652266 (+ 459us) server_negotiation.cc:407] Connection header received
0924 11:48:10.652440 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.652444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.652499 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.652585 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:10.654108 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.654839 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.656681 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.657054 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.657205 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.661337 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:10.661343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:10.661348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:10.661399 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:10.664512 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.665183 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.665188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.665192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.665312 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:10.665645 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:10.665650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:10.665653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:10.665928 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:10.666066 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.666451 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.666677 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260924 11:48:10.671795 9443 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:58650:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:48:10.672807 9443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:10.679569 9580 tablet_service.cc:1467] Processing CreateTablet for tablet 207360880b96466cae334a967ad8939a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89779862ff914b2886de56e7753d3f74]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:48:10.679821 9858 tablet_service.cc:1467] Processing CreateTablet for tablet 207360880b96466cae334a967ad8939a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89779862ff914b2886de56e7753d3f74]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:48:10.680146 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207360880b96466cae334a967ad8939a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.680325 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207360880b96466cae334a967ad8939a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.680955 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 207360880b96466cae334a967ad8939a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89779862ff914b2886de56e7753d3f74]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:48:10.681476 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207360880b96466cae334a967ad8939a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:10.687250 9953 tablet_bootstrap.cc:492] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: Bootstrap starting.
I20260924 11:48:10.689193 9953 tablet_bootstrap.cc:654] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: Neither blocks nor log segments found. Creating new log.
W20260924 11:48:10.690770 9929 tablet.cc:2369] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:48:10.694332 9790 tablet.cc:2369] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:48:10.695885 9952 tablet_bootstrap.cc:492] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: Bootstrap starting.
I20260924 11:48:10.697505 9952 tablet_bootstrap.cc:654] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.705341 9954 tablet_bootstrap.cc:492] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: Bootstrap starting.
I20260924 11:48:10.706818 9953 tablet_bootstrap.cc:492] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: No bootstrap required, opened a new log
I20260924 11:48:10.707044 9953 ts_tablet_manager.cc:1397] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.011s
I20260924 11:48:10.707094 9954 tablet_bootstrap.cc:654] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:10.708076 9953 raft_consensus.cc:348] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.708379 9953 raft_consensus.cc:374] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.709568 9953 raft_consensus.cc:729] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73442790fe5145c58a43cd75d600871a, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.709956 9953 consensus_queue.cc:260] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.710449 9959 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.710830 9959 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.713650 9953 ts_tablet_manager.cc:1428] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260924 11:48:10.717893 9952 tablet_bootstrap.cc:492] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: No bootstrap required, opened a new log
I20260924 11:48:10.718067 9952 ts_tablet_manager.cc:1397] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.012s
I20260924 11:48:10.718845 9952 raft_consensus.cc:348] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.719074 9952 raft_consensus.cc:374] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.719146 9952 raft_consensus.cc:729] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fe945aa114939bc98346691650908, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.719388 9954 tablet_bootstrap.cc:492] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: No bootstrap required, opened a new log
I20260924 11:48:10.719427 9952 consensus_queue.cc:260] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.719609 9954 ts_tablet_manager.cc:1397] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260924 11:48:10.719887 9958 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.720191 9958 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.720577 9954 raft_consensus.cc:348] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.720893 9954 raft_consensus.cc:374] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:10.721019 9954 raft_consensus.cc:729] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c777ec3504384b6db187df4487ada918, State: Initialized, Role: FOLLOWER
I20260924 11:48:10.721340 9954 consensus_queue.cc:260] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.721889 9971 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.722065 9954 ts_tablet_manager.cc:1428] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:10.722208 9971 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.722625 9952 ts_tablet_manager.cc:1428] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 11:48:10.894750 9971 raft_consensus.cc:482] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:10.894959 9971 raft_consensus.cc:504] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.895592 9971 leader_election.cc:288] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:10.896149 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207360880b96466cae334a967ad8939a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a" is_pre_election: true
I20260924 11:48:10.896257 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207360880b96466cae334a967ad8939a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908" is_pre_election: true
I20260924 11:48:10.896423 9600 raft_consensus.cc:2454] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:10.896555 9744 raft_consensus.cc:2454] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:10.897013 9811 leader_election.cc:302] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60fe945aa114939bc98346691650908, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:10.897367 9971 raft_consensus.cc:2781] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:10.897480 9971 raft_consensus.cc:482] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:10.897593 9971 raft_consensus.cc:3037] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.900458 9971 raft_consensus.cc:504] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.901094 9971 leader_election.cc:288] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Requested vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:10.901496 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207360880b96466cae334a967ad8939a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a"
I20260924 11:48:10.901539 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207360880b96466cae334a967ad8939a" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908"
I20260924 11:48:10.901717 9600 raft_consensus.cc:3037] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.901777 9744 raft_consensus.cc:3037] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:10.903658 9744 raft_consensus.cc:2454] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:10.904048 9811 leader_election.cc:302] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60fe945aa114939bc98346691650908, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:10.904314 9971 raft_consensus.cc:2781] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:10.904430 9600 raft_consensus.cc:2454] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:10.904485 9971 raft_consensus.cc:686] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Becoming Leader. State: Replica: c777ec3504384b6db187df4487ada918, State: Running, Role: LEADER
I20260924 11:48:10.904724 9971 consensus_queue.cc:237] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:10.906311 9960 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: TxnStatusTablet state changed. Reason: New leader c777ec3504384b6db187df4487ada918. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.906550 9960 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:48:10.907126 9992 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:48:10.907500 9992 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:48:10.908391 9443 catalog_manager.cc:5495] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 reported cstate change: term changed from 0 to 1, leader changed from <none> to c777ec3504384b6db187df4487ada918 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } health_report { overall_health: HEALTHY } } }
I20260924 11:48:10.953465 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.947264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.1.210.3:34987)
0924 11:48:10.948303 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.948308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.948323 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:48:10.948381 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.948384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.948441 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.948521 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:10.948969 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.949790 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.951683 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.952121 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.952212 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.952728 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:10.952838 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:10.952900 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.953151 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.953213 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":796,"threads_started":1}
I20260924 11:48:10.953575 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.947184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:34987 (local address 127.0.0.1:39788)
0924 11:48:10.947442 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.947469 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.947684 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.948603 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.948608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.948623 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:48:10.948827 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.948837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.949922 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.949928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.951546 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.951561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.952261 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.952267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.952479 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.952553 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:48:10.952932 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:48:10.952952 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.953115 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.953265 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:48:10.957502 9600 raft_consensus.cc:1261] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:10.957502 9744 raft_consensus.cc:1261] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:10.958091 9960 consensus_queue.cc:1035] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:10.958416 9971 consensus_queue.cc:1035] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:10.966066 9959 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: TxnStatusTablet state changed. Reason: New leader c777ec3504384b6db187df4487ada918. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.966361 9959 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.968739 9958 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: TxnStatusTablet state changed. Reason: New leader c777ec3504384b6db187df4487ada918. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.969010 9958 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.969350 9995 mvcc.cc:205] Tried to move back new op lower bound from 7332866010955911168 to 7332866010748141568. Current Snapshot: MvccSnapshot[applied={T|T < 7332866010955911168}]
I20260924 11:48:10.969941 9960 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.970242 9960 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:48:10.970438 9996 mvcc.cc:205] Tried to move back new op lower bound from 7332866010955911168 to 7332866010748141568. Current Snapshot: MvccSnapshot[applied={T|T < 7332866010955911168}]
I20260924 11:48:10.973487 9959 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.973704 9959 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P 73442790fe5145c58a43cd75d600871a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.974397 9971 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.974714 9971 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P c777ec3504384b6db187df4487ada918: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:48:10.979847 9958 tablet_replica.cc:439] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } } }
I20260924 11:48:10.980077 9958 tablet_replica.cc:442] T 207360880b96466cae334a967ad8939a P a60fe945aa114939bc98346691650908: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:48:10.992913 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.986500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:55227 (local address 127.1.210.3:34987)
0924 11:48:10.986679 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:10.986685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:10.987259 (+ 574us) server_negotiation.cc:407] Connection header received
0924 11:48:10.987416 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:10.987421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:10.987519 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:10.987619 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:10.988122 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.989383 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.990992 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.991717 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.991847 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.992200 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:10.992321 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:10.992415 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:10.992525 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 11:48:10.992606 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:48:10.992930 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:10.986472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:34987 (local address 127.1.210.3:55227)
0924 11:48:10.987164 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:10.987180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:10.987306 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:10.987631 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:10.987634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:10.987643 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:48:10.987981 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.987991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.989527 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.989532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.990872 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:10.990883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:10.991868 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:10.991872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:10.992001 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:10.992048 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:48:10.992417 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:48:10.992424 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:48:10.992482 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 11:48:10.992575 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":368,"threads_started":1}
I20260924 11:48:11.001935 9744 raft_consensus.cc:1261] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:11.002106 9599 raft_consensus.cc:1261] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:11.002431 9971 consensus_queue.cc:1035] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:11.002661 9960 consensus_queue.cc:1035] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:11.070554 9441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:11.070824 9441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:11.073819 9441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Sending DeleteTablet for 3 replicas of tablet 62337ec77c5a41dfa93060c012367f63
I20260924 11:48:11.074518 9580 tablet_service.cc:1514] Processing DeleteTablet for tablet 62337ec77c5a41dfa93060c012367f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:59984
I20260924 11:48:11.074610 9719 tablet_service.cc:1514] Processing DeleteTablet for tablet 62337ec77c5a41dfa93060c012367f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:45058
I20260924 11:48:11.075263 10013 tablet_replica.cc:330] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: stopping tablet replica
I20260924 11:48:11.075222 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet 62337ec77c5a41dfa93060c012367f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:39762
I20260924 11:48:11.075564 10013 raft_consensus.cc:2229] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:11.075857 10013 raft_consensus.cc:2258] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.076916 10013 ts_tablet_manager.cc:1905] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.084331 10013 ts_tablet_manager.cc:1918] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:48:11.084542 10013 log.cc:1196] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/62337ec77c5a41dfa93060c012367f63
I20260924 11:48:11.085129 10013 ts_tablet_manager.cc:1939] T 62337ec77c5a41dfa93060c012367f63 P 73442790fe5145c58a43cd75d600871a: Deleting consensus metadata
I20260924 11:48:11.086292 9429 catalog_manager.cc:4841] TS 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403): tablet 62337ec77c5a41dfa93060c012367f63 (table test-table [id=8b4def6e8cee4fc48ca5ab87f8e03fab]) successfully deleted
I20260924 11:48:11.089321 10014 tablet_replica.cc:330] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: stopping tablet replica
I20260924 11:48:11.089600 10014 raft_consensus.cc:2229] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:11.089989 10014 raft_consensus.cc:2258] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.091454 10014 ts_tablet_manager.cc:1905] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.098604 10014 ts_tablet_manager.cc:1918] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:48:11.099205 10015 tablet_replica.cc:330] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: stopping tablet replica
I20260924 11:48:11.099434 10015 raft_consensus.cc:2229] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:11.099895 10015 raft_consensus.cc:2258] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.100209 10014 log.cc:1196] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/62337ec77c5a41dfa93060c012367f63
I20260924 11:48:11.100765 10014 ts_tablet_manager.cc:1939] T 62337ec77c5a41dfa93060c012367f63 P a60fe945aa114939bc98346691650908: Deleting consensus metadata
I20260924 11:48:11.101040 10015 ts_tablet_manager.cc:1905] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.101974 9429 catalog_manager.cc:4841] TS a60fe945aa114939bc98346691650908 (127.1.210.2:35375): tablet 62337ec77c5a41dfa93060c012367f63 (table test-table [id=8b4def6e8cee4fc48ca5ab87f8e03fab]) successfully deleted
I20260924 11:48:11.108816 10015 ts_tablet_manager.cc:1918] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:48:11.108971 10015 log.cc:1196] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/62337ec77c5a41dfa93060c012367f63
I20260924 11:48:11.109426 10015 ts_tablet_manager.cc:1939] T 62337ec77c5a41dfa93060c012367f63 P c777ec3504384b6db187df4487ada918: Deleting consensus metadata
I20260924 11:48:11.110328 9428 catalog_manager.cc:4841] TS c777ec3504384b6db187df4487ada918 (127.1.210.3:34987): tablet 62337ec77c5a41dfa93060c012367f63 (table test-table [id=8b4def6e8cee4fc48ca5ab87f8e03fab]) successfully deleted
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:48:11.144160 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:11.132319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.1.210.62:37493)
0924 11:48:11.132563 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:11.132569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:11.132929 (+ 360us) server_negotiation.cc:407] Connection header received
0924 11:48:11.133167 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:11.133171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:11.133235 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:11.133320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:11.135130 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.135823 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.137828 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.138242 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.138338 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:11.143873 (+ 5535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58666: BlockingRecv error: recv got EOF from 127.0.0.1:58666 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM
I20260924 11:48:11.162735 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:11.147038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.1.210.62:37493)
0924 11:48:11.147245 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:11.147250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:11.147536 (+ 286us) server_negotiation.cc:407] Connection header received
0924 11:48:11.147711 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:11.147714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:11.147772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:11.147853 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:11.149476 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.150159 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.151984 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.152381 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.152471 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:11.157153 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:11.157158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:11.157161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:11.157193 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:11.159929 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.160739 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.160745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.160749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.160926 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.161362 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.161366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.161370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.161671 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:11.161862 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:11.162247 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 11:48:11.162423 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:48:11.166146 9441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:11.166677 9441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:11.173162 9580 tablet_service.cc:1467] Processing CreateTablet for tablet 8d42cba6c303490b90cc909860c91934 (DEFAULT_TABLE table=test-table [id=bb4383994a764eb3b329682d1a2957e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:11.173604 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d42cba6c303490b90cc909860c91934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:11.173502 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 8d42cba6c303490b90cc909860c91934 (DEFAULT_TABLE table=test-table [id=bb4383994a764eb3b329682d1a2957e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:11.173894 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d42cba6c303490b90cc909860c91934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:11.177600 9858 tablet_service.cc:1467] Processing CreateTablet for tablet 8d42cba6c303490b90cc909860c91934 (DEFAULT_TABLE table=test-table [id=bb4383994a764eb3b329682d1a2957e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:11.178079 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d42cba6c303490b90cc909860c91934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:11.181859 9952 tablet_bootstrap.cc:492] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Bootstrap starting.
I20260924 11:48:11.181936 9953 tablet_bootstrap.cc:492] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Bootstrap starting.
I20260924 11:48:11.183399 9952 tablet_bootstrap.cc:654] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:11.183786 9953 tablet_bootstrap.cc:654] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:11.193364 9954 tablet_bootstrap.cc:492] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Bootstrap starting.
I20260924 11:48:11.193977 9953 tablet_bootstrap.cc:492] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: No bootstrap required, opened a new log
I20260924 11:48:11.194100 9953 ts_tablet_manager.cc:1397] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260924 11:48:11.195183 9954 tablet_bootstrap.cc:654] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:11.194785 9953 raft_consensus.cc:348] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.195291 9953 raft_consensus.cc:374] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:11.195382 9953 raft_consensus.cc:729] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73442790fe5145c58a43cd75d600871a, State: Initialized, Role: FOLLOWER
I20260924 11:48:11.195609 9953 consensus_queue.cc:260] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.196702 9953 ts_tablet_manager.cc:1428] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 11:48:11.198225 9954 tablet_bootstrap.cc:492] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: No bootstrap required, opened a new log
I20260924 11:48:11.198429 9954 ts_tablet_manager.cc:1397] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260924 11:48:11.199301 9954 raft_consensus.cc:348] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.199656 9954 raft_consensus.cc:374] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:11.199810 9954 raft_consensus.cc:729] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c777ec3504384b6db187df4487ada918, State: Initialized, Role: FOLLOWER
I20260924 11:48:11.200212 9954 consensus_queue.cc:260] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.200850 9954 ts_tablet_manager.cc:1428] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:11.202073 9952 tablet_bootstrap.cc:492] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: No bootstrap required, opened a new log
I20260924 11:48:11.202221 9952 ts_tablet_manager.cc:1397] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20260924 11:48:11.202971 9952 raft_consensus.cc:348] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.203236 9952 raft_consensus.cc:374] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:11.203330 9952 raft_consensus.cc:729] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fe945aa114939bc98346691650908, State: Initialized, Role: FOLLOWER
I20260924 11:48:11.203526 9952 consensus_queue.cc:260] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.204027 9952 ts_tablet_manager.cc:1428] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 11:48:11.226872 10009 raft_consensus.cc:482] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:11.227056 10009 raft_consensus.cc:504] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.227815 10009 leader_election.cc:288] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:11.228322 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42cba6c303490b90cc909860c91934" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a" is_pre_election: true
I20260924 11:48:11.228396 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42cba6c303490b90cc909860c91934" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908" is_pre_election: true
I20260924 11:48:11.228566 9599 raft_consensus.cc:2454] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:11.228690 9744 raft_consensus.cc:2454] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:11.229017 9811 leader_election.cc:302] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73442790fe5145c58a43cd75d600871a, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:11.229408 10009 raft_consensus.cc:2781] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:11.229602 10009 raft_consensus.cc:482] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:11.229724 10009 raft_consensus.cc:3037] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:11.232867 10009 raft_consensus.cc:504] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.233501 10009 leader_election.cc:288] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Requested vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:11.234009 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42cba6c303490b90cc909860c91934" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a"
I20260924 11:48:11.234215 9599 raft_consensus.cc:3037] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:11.234237 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42cba6c303490b90cc909860c91934" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908"
I20260924 11:48:11.234437 9744 raft_consensus.cc:3037] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:11.236924 9599 raft_consensus.cc:2454] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:11.237419 9811 leader_election.cc:302] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73442790fe5145c58a43cd75d600871a, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:11.237939 9744 raft_consensus.cc:2454] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:11.238312 10009 raft_consensus.cc:2781] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:11.238632 10009 raft_consensus.cc:686] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Becoming Leader. State: Replica: c777ec3504384b6db187df4487ada918, State: Running, Role: LEADER
I20260924 11:48:11.238978 10009 consensus_queue.cc:237] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.245959 9443 catalog_manager.cc:5495] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 reported cstate change: term changed from 0 to 1, leader changed from <none> to c777ec3504384b6db187df4487ada918 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } health_report { overall_health: HEALTHY } } }
I20260924 11:48:11.262162 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:11.255878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.1.210.3:34987)
0924 11:48:11.256074 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:11.256081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:11.256399 (+ 318us) server_negotiation.cc:407] Connection header received
0924 11:48:11.256596 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:11.256600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:11.256660 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:11.256734 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:11.257178 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.257930 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.260058 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.260669 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.260813 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:11.261415 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:11.261537 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:11.261625 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:11.261782 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 11:48:11.261845 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:48:11.264669 9599 raft_consensus.cc:1261] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:11.264796 9744 raft_consensus.cc:1261] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:11.265374 10009 consensus_queue.cc:1035] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:11.265702 9971 consensus_queue.cc:1035] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:11.279417 9443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:11.280503 9443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:11.283888 9443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Sending DeleteTablet for 3 replicas of tablet 8d42cba6c303490b90cc909860c91934
I20260924 11:48:11.284878 9580 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42cba6c303490b90cc909860c91934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:59984
I20260924 11:48:11.284967 9719 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42cba6c303490b90cc909860c91934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:45058
I20260924 11:48:11.285197 10013 tablet_replica.cc:330] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: stopping tablet replica
I20260924 11:48:11.285260 10014 tablet_replica.cc:330] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: stopping tablet replica
I20260924 11:48:11.285224 9579 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42cba6c303490b90cc909860c91934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:59984
I20260924 11:48:11.285431 10014 raft_consensus.cc:2229] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:11.285458 10013 raft_consensus.cc:2229] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:11.285621 10014 raft_consensus.cc:2258] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.285708 10013 raft_consensus.cc:2258] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.285995 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42cba6c303490b90cc909860c91934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:39762
I20260924 11:48:11.286288 10015 tablet_replica.cc:330] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: stopping tablet replica
I20260924 11:48:11.286499 10015 raft_consensus.cc:2229] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:11.286681 10014 ts_tablet_manager.cc:1905] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.286813 10015 raft_consensus.cc:2258] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.287186 9443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:11.287773 9443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:11.289568 10013 ts_tablet_manager.cc:1905] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.294895 10015 ts_tablet_manager.cc:1905] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.297154 10013 ts_tablet_manager.cc:1918] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:11.297384 10013 log.cc:1196] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/8d42cba6c303490b90cc909860c91934
I20260924 11:48:11.297809 10014 ts_tablet_manager.cc:1918] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:11.297951 10014 log.cc:1196] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/8d42cba6c303490b90cc909860c91934
I20260924 11:48:11.298076 10013 ts_tablet_manager.cc:1939] T 8d42cba6c303490b90cc909860c91934 P 73442790fe5145c58a43cd75d600871a: Deleting consensus metadata
I20260924 11:48:11.298362 10014 ts_tablet_manager.cc:1939] T 8d42cba6c303490b90cc909860c91934 P a60fe945aa114939bc98346691650908: Deleting consensus metadata
I20260924 11:48:11.299259 9429 catalog_manager.cc:4841] TS 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403): tablet 8d42cba6c303490b90cc909860c91934 (table test-table [id=bb4383994a764eb3b329682d1a2957e9]) successfully deleted
I20260924 11:48:11.299551 9429 catalog_manager.cc:4841] TS a60fe945aa114939bc98346691650908 (127.1.210.2:35375): tablet 8d42cba6c303490b90cc909860c91934 (table test-table [id=bb4383994a764eb3b329682d1a2957e9]) successfully deleted
W20260924 11:48:11.300640 9429 catalog_manager.cc:4808] TS 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403): delete failed for tablet 8d42cba6c303490b90cc909860c91934 because the tablet was not found. No further retry: Not found: Tablet not found: 8d42cba6c303490b90cc909860c91934
I20260924 11:48:11.304346 10015 ts_tablet_manager.cc:1918] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:11.304596 10015 log.cc:1196] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/8d42cba6c303490b90cc909860c91934
I20260924 11:48:11.305194 10015 ts_tablet_manager.cc:1939] T 8d42cba6c303490b90cc909860c91934 P c777ec3504384b6db187df4487ada918: Deleting consensus metadata
I20260924 11:48:11.305371 9579 tablet_service.cc:1467] Processing CreateTablet for tablet 920ae34ff98a426e98684d23ff6603f7 (DEFAULT_TABLE table=test-table [id=31d75018c51e42d1a239b53b3a2e589f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:11.305809 9579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920ae34ff98a426e98684d23ff6603f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:11.308279 9428 catalog_manager.cc:4841] TS c777ec3504384b6db187df4487ada918 (127.1.210.3:34987): tablet 8d42cba6c303490b90cc909860c91934 (table test-table [id=bb4383994a764eb3b329682d1a2957e9]) successfully deleted
I20260924 11:48:11.308454 9858 tablet_service.cc:1467] Processing CreateTablet for tablet 920ae34ff98a426e98684d23ff6603f7 (DEFAULT_TABLE table=test-table [id=31d75018c51e42d1a239b53b3a2e589f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:11.308902 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920ae34ff98a426e98684d23ff6603f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:11.308964 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 920ae34ff98a426e98684d23ff6603f7 (DEFAULT_TABLE table=test-table [id=31d75018c51e42d1a239b53b3a2e589f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:11.309401 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920ae34ff98a426e98684d23ff6603f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:11.311365 9953 tablet_bootstrap.cc:492] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Bootstrap starting.
I20260924 11:48:11.312964 9953 tablet_bootstrap.cc:654] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:11.315397 9954 tablet_bootstrap.cc:492] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Bootstrap starting.
I20260924 11:48:11.315747 9952 tablet_bootstrap.cc:492] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Bootstrap starting.
I20260924 11:48:11.317211 9954 tablet_bootstrap.cc:654] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:11.317488 9952 tablet_bootstrap.cc:654] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:11.329592 9952 tablet_bootstrap.cc:492] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: No bootstrap required, opened a new log
I20260924 11:48:11.329771 9952 ts_tablet_manager.cc:1397] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260924 11:48:11.330066 9954 tablet_bootstrap.cc:492] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: No bootstrap required, opened a new log
I20260924 11:48:11.330233 9954 ts_tablet_manager.cc:1397] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260924 11:48:11.330672 9952 raft_consensus.cc:348] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.330936 9952 raft_consensus.cc:374] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:11.331043 9952 raft_consensus.cc:729] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60fe945aa114939bc98346691650908, State: Initialized, Role: FOLLOWER
I20260924 11:48:11.331120 9954 raft_consensus.cc:348] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.331411 9954 raft_consensus.cc:374] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:11.331341 9952 consensus_queue.cc:260] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.331502 9954 raft_consensus.cc:729] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c777ec3504384b6db187df4487ada918, State: Initialized, Role: FOLLOWER
I20260924 11:48:11.331765 9954 consensus_queue.cc:260] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.332002 9952 ts_tablet_manager.cc:1428] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 11:48:11.332405 9954 ts_tablet_manager.cc:1428] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 11:48:11.334069 9953 tablet_bootstrap.cc:492] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: No bootstrap required, opened a new log
I20260924 11:48:11.334273 9953 ts_tablet_manager.cc:1397] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.012s
I20260924 11:48:11.335157 9953 raft_consensus.cc:348] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.335428 9953 raft_consensus.cc:374] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:11.335520 9953 raft_consensus.cc:729] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73442790fe5145c58a43cd75d600871a, State: Initialized, Role: FOLLOWER
I20260924 11:48:11.335827 9953 consensus_queue.cc:260] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.336404 9953 ts_tablet_manager.cc:1428] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 11:48:11.441246 10009 raft_consensus.cc:482] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:11.441490 10009 raft_consensus.cc:504] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.442157 10009 leader_election.cc:288] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:11.442729 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ae34ff98a426e98684d23ff6603f7" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a" is_pre_election: true
I20260924 11:48:11.442883 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ae34ff98a426e98684d23ff6603f7" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908" is_pre_election: true
I20260924 11:48:11.443013 9599 raft_consensus.cc:2454] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:11.443150 9744 raft_consensus.cc:2454] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 0.
I20260924 11:48:11.443601 9811 leader_election.cc:302] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73442790fe5145c58a43cd75d600871a, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:11.443867 10009 raft_consensus.cc:2781] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:11.444001 10009 raft_consensus.cc:482] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:11.444124 10009 raft_consensus.cc:3037] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:11.446287 10009 raft_consensus.cc:504] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.446844 10009 leader_election.cc:288] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Requested vote from peers 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403), a60fe945aa114939bc98346691650908 (127.1.210.2:35375)
I20260924 11:48:11.447332 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ae34ff98a426e98684d23ff6603f7" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73442790fe5145c58a43cd75d600871a"
I20260924 11:48:11.447376 9744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ae34ff98a426e98684d23ff6603f7" candidate_uuid: "c777ec3504384b6db187df4487ada918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60fe945aa114939bc98346691650908"
I20260924 11:48:11.447582 9744 raft_consensus.cc:3037] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:11.447595 9599 raft_consensus.cc:3037] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:11.449673 9744 raft_consensus.cc:2454] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:11.449782 9599 raft_consensus.cc:2454] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c777ec3504384b6db187df4487ada918 in term 1.
I20260924 11:48:11.450096 9811 leader_election.cc:302] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60fe945aa114939bc98346691650908, c777ec3504384b6db187df4487ada918; no voters:
I20260924 11:48:11.450416 10009 raft_consensus.cc:2781] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:11.450590 10009 raft_consensus.cc:686] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Becoming Leader. State: Replica: c777ec3504384b6db187df4487ada918, State: Running, Role: LEADER
I20260924 11:48:11.450922 10009 consensus_queue.cc:237] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } }
I20260924 11:48:11.453434 9443 catalog_manager.cc:5495] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 reported cstate change: term changed from 0 to 1, leader changed from <none> to c777ec3504384b6db187df4487ada918 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "c777ec3504384b6db187df4487ada918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c777ec3504384b6db187df4487ada918" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 34987 } health_report { overall_health: HEALTHY } } }
I20260924 11:48:11.483343 9600 raft_consensus.cc:1261] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:11.483352 9744 raft_consensus.cc:1261] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Refusing update from remote peer c777ec3504384b6db187df4487ada918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:11.484292 10006 consensus_queue.cc:1035] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73442790fe5145c58a43cd75d600871a" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:11.484558 10009 consensus_queue.cc:1035] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60fe945aa114939bc98346691650908" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:11.524267 9443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:11.524539 9443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:11.527321 9443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 403fa00ab15243ccaa845eedd1e376e8: Sending DeleteTablet for 3 replicas of tablet 920ae34ff98a426e98684d23ff6603f7
I20260924 11:48:11.528021 9579 tablet_service.cc:1514] Processing DeleteTablet for tablet 920ae34ff98a426e98684d23ff6603f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:59984
I20260924 11:48:11.528095 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet 920ae34ff98a426e98684d23ff6603f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:39762
I20260924 11:48:11.528330 10013 tablet_replica.cc:330] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: stopping tablet replica
I20260924 11:48:11.528352 10015 tablet_replica.cc:330] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: stopping tablet replica
I20260924 11:48:11.528514 10015 raft_consensus.cc:2229] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:11.528528 10013 raft_consensus.cc:2229] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:11.528720 10013 raft_consensus.cc:2258] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.528777 10015 raft_consensus.cc:2258] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.529342 9719 tablet_service.cc:1514] Processing DeleteTablet for tablet 920ae34ff98a426e98684d23ff6603f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:11 UTC) from {username='oryx', principal='oryx/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:45058
I20260924 11:48:11.529623 10014 tablet_replica.cc:330] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: stopping tablet replica
I20260924 11:48:11.529788 10014 raft_consensus.cc:2229] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:11.530005 10014 raft_consensus.cc:2258] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:11.530524 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 9514
I20260924 11:48:11.530952 10015 ts_tablet_manager.cc:1905] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.531329 10013 ts_tablet_manager.cc:1905] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.535550 10015 ts_tablet_manager.cc:1918] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:48:11.535683 10015 log.cc:1196] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/920ae34ff98a426e98684d23ff6603f7
I20260924 11:48:11.536919 10014 ts_tablet_manager.cc:1905] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:11.538204 10013 ts_tablet_manager.cc:1918] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:48:11.538324 10013 log.cc:1196] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/920ae34ff98a426e98684d23ff6603f7
I20260924 11:48:11.538769 10013 ts_tablet_manager.cc:1939] T 920ae34ff98a426e98684d23ff6603f7 P 73442790fe5145c58a43cd75d600871a: Deleting consensus metadata
I20260924 11:48:11.539487 10015 ts_tablet_manager.cc:1939] T 920ae34ff98a426e98684d23ff6603f7 P c777ec3504384b6db187df4487ada918: Deleting consensus metadata
I20260924 11:48:11.539867 9429 catalog_manager.cc:4841] TS 73442790fe5145c58a43cd75d600871a (127.1.210.1:46403): tablet 920ae34ff98a426e98684d23ff6603f7 (table test-table [id=31d75018c51e42d1a239b53b3a2e589f]) successfully deleted
I20260924 11:48:11.540431 9428 catalog_manager.cc:4841] TS c777ec3504384b6db187df4487ada918 (127.1.210.3:34987): tablet 920ae34ff98a426e98684d23ff6603f7 (table test-table [id=31d75018c51e42d1a239b53b3a2e589f]) successfully deleted
I20260924 11:48:11.543634 10014 ts_tablet_manager.cc:1918] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:48:11.543789 10014 log.cc:1196] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/920ae34ff98a426e98684d23ff6603f7
I20260924 11:48:11.544191 10014 ts_tablet_manager.cc:1939] T 920ae34ff98a426e98684d23ff6603f7 P a60fe945aa114939bc98346691650908: Deleting consensus metadata
I20260924 11:48:11.545113 9429 catalog_manager.cc:4841] TS a60fe945aa114939bc98346691650908 (127.1.210.2:35375): tablet 920ae34ff98a426e98684d23ff6603f7 (table test-table [id=31d75018c51e42d1a239b53b3a2e589f]) successfully deleted
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.1@KRBTEST.COM
I20260924 11:48:11.555527 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:11.536039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.1.210.1:46403)
0924 11:48:11.540549 (+ 4510us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:11.540556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:11.541669 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 11:48:11.541734 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:11.541740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:11.541846 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:11.541962 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:11.543493 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.544651 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.546048 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.546712 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.546819 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:11.550457 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:11.550471 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:11.550487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:11.550516 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:11.553430 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.553985 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.553992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.553995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.554103 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.554410 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.554413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.554415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.554721 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:11.554946 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:11.555135 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:48:11.555284 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4387,"thread_start_us":4532,"threads_started":1}
I20260924 11:48:11.556150 9645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:11.651926 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 9514
I20260924 11:48:11.670270 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 9654
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.2@KRBTEST.COM
I20260924 11:48:11.684866 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:11.671066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.1.210.2:35375)
0924 11:48:11.671531 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:11.671536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:11.671552 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:48:11.671604 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:11.671608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:11.671671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:11.671774 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:11.672905 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.673721 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.674881 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.675261 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.675350 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:11.679298 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:11.679312 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:11.679328 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:11.679360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:11.682448 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.683114 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.683121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.683124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.683260 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.683557 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.683559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.683561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.683888 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:11.684095 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:11.684309 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 11:48:11.684475 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":251,"threads_started":1}
I20260924 11:48:11.685587 9784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:11.780936 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 9654
I20260924 11:48:11.798827 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 9793
Sep 24 11:48:11 dist-test-slave-dd32 krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.3@KRBTEST.COM
I20260924 11:48:11.813747 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:11.799609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.1.210.3:34987)
0924 11:48:11.800079 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:11.800084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:11.800099 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:48:11.800150 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:11.800154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:11.800209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:11.800313 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:11.801617 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.802299 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.803468 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:11.803843 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:11.803930 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:11.807922 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:11.807937 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:11.807953 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:11.808003 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:11.811231 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.811846 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.811852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.811855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.812003 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:11.812319 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:11.812322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:11.812324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:11.812669 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:11.812889 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:11.813153 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:48:11.813341 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":277,"threads_started":1}
I20260924 11:48:11.814417 9923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:11.955646 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 9793
I20260924 11:48:11.980049 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 9411
I20260924 11:48:11.981199 9483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:12.093806 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 9411
2026-09-24T11:48:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5298 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:48:12 dist-test-slave-dd32 krb5kdc[10052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:48:12 dist-test-slave-dd32 krb5kdc[10052](info): set up 2 sockets
Sep 24 11:48:12 dist-test-slave-dd32 krb5kdc[10052](info): commencing operation
krb5kdc: starting...
W20260924 11:48:13.194205 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:48:13 dist-test-slave-dd32 krb5kdc[10052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:48:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:48:13.396576 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:44527
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:13.707621 10065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:13.707962 10065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:13.708137 10065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:13.715212 10065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:13.715344 10065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:13.715459 10065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:13.715560 10065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:13.715647 10065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:13.725116 10065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:44527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:13.726395 10065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:13.727669 10065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:13.739418 10070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:13.739377 10071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:13.739442 10073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:13.740513 10065 server_base.cc:1034] running on GCE node
I20260924 11:48:13.741307 10065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:13.742601 10065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:13.743825 10065 hybrid_clock.cc:648] HybridClock initialized: now 1790250493743775 us; error 35 us; skew 500 ppm
I20260924 11:48:13.744148 10065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T11:48:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1716 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:48:13 dist-test-slave-dd32 krb5kdc[10079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:48:13 dist-test-slave-dd32 krb5kdc[10079](info): set up 2 sockets
Sep 24 11:48:13 dist-test-slave-dd32 krb5kdc[10079](info): commencing operation
krb5kdc: starting...
W20260924 11:48:14.906111 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:48:15 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:48:15.114768 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:45337
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:15.428654 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:15.429004 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:15.429167 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:15.436188 10092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:15.436322 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:15.436450 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:15.436555 10092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:15.436632 10092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:15.446319 10092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:15.447603 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:15.448925 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:15.460418 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:15.460371 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:15.460455 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:15.461267 10092 server_base.cc:1034] running on GCE node
I20260924 11:48:15.462100 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:15.463480 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:15.464706 10092 hybrid_clock.cc:648] HybridClock initialized: now 1790250495464675 us; error 58 us; skew 500 ppm
I20260924 11:48:15.465093 10092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:15 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:15.467998 10092 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:48:15.469470 10092 webserver.cc:460] Webserver started at http://127.1.210.62:41635/ using document root <none> and password file <none>
I20260924 11:48:15.470122 10092 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:15.470242 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:15.470568 10092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:15.473479 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1"
format_stamp: "Formatted at 2026-09-24 11:48:15 on dist-test-slave-dd32"
server_key: "e641205154e72c2eb321cce3cb6e6daf"
server_key_iv: "de9711d35b73a02b821777df9b0370bb"
server_key_version: "encryptionkey@0"
I20260924 11:48:15.474231 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1"
format_stamp: "Formatted at 2026-09-24 11:48:15 on dist-test-slave-dd32"
server_key: "e641205154e72c2eb321cce3cb6e6daf"
server_key_iv: "de9711d35b73a02b821777df9b0370bb"
server_key_version: "encryptionkey@0"
I20260924 11:48:15.479024 10092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:15.482184 10107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:15.483433 10092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:15.483609 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1"
format_stamp: "Formatted at 2026-09-24 11:48:15 on dist-test-slave-dd32"
server_key: "e641205154e72c2eb321cce3cb6e6daf"
server_key_iv: "de9711d35b73a02b821777df9b0370bb"
server_key_version: "encryptionkey@0"
I20260924 11:48:15.483791 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:15.500625 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:15.504300 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:15.504603 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:15.522027 10092 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:45337
I20260924 11:48:15.522047 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:45337 every 8 connection(s)
I20260924 11:48:15.523380 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:48:15.523882 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10092
I20260924 11:48:15.524111 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:48:15.524962 1864 external_mini_cluster.cc:1428] Setting key cc6b0a7b7ecd0604990be6c9e1444785
I20260924 11:48:15.527627 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:48:15 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:15.540292 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Bootstrap starting.
I20260924 11:48:15.543890 10159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:15.544902 10159 log.cc:824] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:15.547829 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: No bootstrap required, opened a new log
I20260924 11:48:15.549582 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:15.526913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.1.210.62:45337)
0924 11:48:15.527776 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:15.527789 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:15.527829 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:15.529483 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:15.529503 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:15.529862 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:15.530174 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:15.531907 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:15.533305 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:15.535006 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:15.535617 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:15.535744 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:15.540637 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:15.540649 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:15.540670 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:15.540742 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:15.545370 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:15.546346 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:15.546353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:15.546363 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:15.546541 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:15.546975 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:15.546978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:15.546980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:15.547458 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:15.547656 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:15.548155 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 11:48:15.548522 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":383,"threads_started":1}
I20260924 11:48:15.555531 10159 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:15.555928 10159 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:15.556038 10159 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e3b48fc2e04bd1bca574b16f58f2d1, State: Initialized, Role: FOLLOWER
I20260924 11:48:15.556550 10159 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:15.556764 10159 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:48:15.556919 10159 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:48:15.557114 10159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:15.559273 10159 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:15.559737 10159 leader_election.cc:302] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e3b48fc2e04bd1bca574b16f58f2d1; no voters:
I20260924 11:48:15.560281 10159 leader_election.cc:288] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:48:15.560416 10164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:15.561000 10164 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 LEADER]: Becoming Leader. State: Replica: a9e3b48fc2e04bd1bca574b16f58f2d1, State: Running, Role: LEADER
I20260924 11:48:15.561676 10164 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:15.562685 10159 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:15.564581 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } } }
I20260924 11:48:15.564961 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:15.564981 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e3b48fc2e04bd1bca574b16f58f2d1. Latest consensus state: current_term: 1 leader_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } } }
I20260924 11:48:15.565538 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:15.567070 10170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:15.571832 10170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:15.579523 10170 catalog_manager.cc:1347] Generated new cluster ID: 8988f2470ce1448390541c64a0f8f0d2
I20260924 11:48:15.579632 10170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:15.590271 10170 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:48:15.592092 10170 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:15.609613 10170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Generated new TSK 0
I20260924 11:48:15.610664 10170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:48:15.673233 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:45337
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:15.992350 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:15.992733 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:15.992877 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:16.000231 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:16.000353 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:16.000550 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:16.010393 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:16.011667 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:16.012959 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:16.025914 10190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:16.025889 10191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:16.026077 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:16.026705 10185 server_base.cc:1034] running on GCE node
I20260924 11:48:16.027216 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:16.028008 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:16.029153 10185 hybrid_clock.cc:648] HybridClock initialized: now 1790250496029082 us; error 55 us; skew 500 ppm
I20260924 11:48:16.029440 10185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:16 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:16.032378 10185 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:48:16.033666 10185 webserver.cc:460] Webserver started at http://127.1.210.1:33089/ using document root <none> and password file <none>
I20260924 11:48:16.034253 10185 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:16.034407 10185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:16.034659 10185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:16.037346 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "929b4dea8f32472ead3b0ea63ba70c9b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "edaf4fb962efb4e1b4781465e74a9468"
server_key_iv: "3576b5713e1e92011ee7b54bc8d7b022"
server_key_version: "encryptionkey@0"
I20260924 11:48:16.038048 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "929b4dea8f32472ead3b0ea63ba70c9b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "edaf4fb962efb4e1b4781465e74a9468"
server_key_iv: "3576b5713e1e92011ee7b54bc8d7b022"
server_key_version: "encryptionkey@0"
I20260924 11:48:16.042665 10185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:16.045799 10200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:16.047008 10185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:16.047176 10185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "929b4dea8f32472ead3b0ea63ba70c9b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "edaf4fb962efb4e1b4781465e74a9468"
server_key_iv: "3576b5713e1e92011ee7b54bc8d7b022"
server_key_version: "encryptionkey@0"
I20260924 11:48:16.047369 10185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:16.083141 10185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:16.087387 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:16.087663 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:16.088631 10185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:16.090001 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:16.090101 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:16.090250 10185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:16.090323 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:16.122437 10185 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:43723
I20260924 11:48:16.122519 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:43723 every 8 connection(s)
I20260924 11:48:16.123683 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:16.126108 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10185
I20260924 11:48:16.126310 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:48:16.127127 1864 external_mini_cluster.cc:1428] Setting key c785659348c59ecb9e523e4fcd60be42
Sep 24 11:48:16 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:16.146266 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:16.126323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:35413 (local address 127.1.210.62:45337)
0924 11:48:16.126769 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:16.126773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:16.130180 (+ 3407us) server_negotiation.cc:407] Connection header received
0924 11:48:16.130226 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:16.130229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:16.130308 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:16.130387 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:16.132499 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.133276 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.135714 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.136075 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.136173 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:16.139484 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:16.139493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:16.139500 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:16.139542 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:16.143045 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:16.143726 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:16.143733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:16.143740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:16.143877 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:16.144259 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:16.144267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:16.144273 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:16.144654 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:16.144804 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:16.145535 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 11:48:16.145786 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":236,"threads_started":1}
I20260924 11:48:16.147024 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:16.127098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:35413)
0924 11:48:16.127912 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:16.127968 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:16.129919 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:16.130695 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:16.130710 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:16.131489 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:16.132326 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:16.132353 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.133424 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.133432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:16.135570 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:16.135588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.136308 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.136319 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:16.136537 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:16.137219 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:16.137244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:16.139276 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:16.143180 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:16.143186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:16.143198 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:16.143604 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:16.143981 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:16.143985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:16.143987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:16.144146 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:16.144770 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:16.144775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:16.145159 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 11:48:16.145396 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 11:48:16.145768 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":329,"threads_started":1}
I20260924 11:48:16.148603 10313 heartbeater.cc:347] Connected to a master server at 127.1.210.62:45337
I20260924 11:48:16.148928 10313 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:16.149827 10313 heartbeater.cc:511] Master 127.1.210.62:45337 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:48:16.152138 10124 ts_manager.cc:194] Registered new tserver with Master: 929b4dea8f32472ead3b0ea63ba70c9b (127.1.210.1:43723)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:48:16.153824 10124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:35413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:48:16.179625 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:45337
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:16.507314 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:16.507627 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:16.507812 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:16.514966 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:16.515148 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:16.515336 10319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:16.525254 10319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:16.526511 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:16.527922 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:16.541433 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:16.541358 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:16.541399 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:16.542199 10319 server_base.cc:1034] running on GCE node
I20260924 11:48:16.542888 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:16.543816 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:16.545101 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790250496545028 us; error 52 us; skew 500 ppm
I20260924 11:48:16.545478 10319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:16 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:16.548486 10319 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:48:16.549983 10319 webserver.cc:460] Webserver started at http://127.1.210.2:34417/ using document root <none> and password file <none>
I20260924 11:48:16.550726 10319 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:16.550871 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:16.551215 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:16.554232 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "735f8a385a4e4ff99d4cdac0b971b01b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "3d2a6f02ffa61a7cce67ef7098cf6a6f"
server_key_iv: "b5c9ed719fc05c0d96ee6ab888a69725"
server_key_version: "encryptionkey@0"
I20260924 11:48:16.555032 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "735f8a385a4e4ff99d4cdac0b971b01b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "3d2a6f02ffa61a7cce67ef7098cf6a6f"
server_key_iv: "b5c9ed719fc05c0d96ee6ab888a69725"
server_key_version: "encryptionkey@0"
I20260924 11:48:16.560204 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 11:48:16.563742 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:16.565155 10319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:48:16.565330 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "735f8a385a4e4ff99d4cdac0b971b01b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "3d2a6f02ffa61a7cce67ef7098cf6a6f"
server_key_iv: "b5c9ed719fc05c0d96ee6ab888a69725"
server_key_version: "encryptionkey@0"
I20260924 11:48:16.565523 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:16.580323 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:16.583863 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:16.584138 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:16.584988 10319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:16.586436 10319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:16.586539 10319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:16.586688 10319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:16.586761 10319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:16.618749 10319 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:42633
I20260924 11:48:16.618791 10446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:42633 every 8 connection(s)
I20260924 11:48:16.620020 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:16.628762 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10319
I20260924 11:48:16.628933 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:48:16.629465 1864 external_mini_cluster.cc:1428] Setting key 17004528d58c3056e44dc55ab2e54045
Sep 24 11:48:16 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:16.640082 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:16.622455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:59835 (local address 127.1.210.62:45337)
0924 11:48:16.622620 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:16.622624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:16.623738 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 11:48:16.625615 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:16.625620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:16.625714 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:16.625846 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:16.627226 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.627909 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.630070 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.630696 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.630839 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:16.634595 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:16.634599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:16.634603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:16.634630 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:16.637206 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:16.637882 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:16.637888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:16.637891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:16.638014 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:16.638436 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:16.638441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:16.638445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:16.638691 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:16.638844 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:16.639511 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 11:48:16.639717 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:48:16.640960 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:16.622862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:59835)
0924 11:48:16.623555 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:16.623602 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:16.625331 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:16.626062 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:16.626069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:16.626525 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:16.627057 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:16.627071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.628071 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.628074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:16.629900 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:16.629914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:16.630886 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:16.630891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:16.631085 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:16.631761 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:16.631788 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:16.634382 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:16.637342 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:16.637350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:16.637365 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:16.637756 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:16.638123 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:16.638129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:16.638133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:16.638320 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:16.638824 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:16.638831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:16.639151 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 11:48:16.639417 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 11:48:16.639909 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260924 11:48:16.642266 10447 heartbeater.cc:347] Connected to a master server at 127.1.210.62:45337
I20260924 11:48:16.642561 10447 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:16.643234 10447 heartbeater.cc:511] Master 127.1.210.62:45337 requested a full tablet report, sending...
I20260924 11:48:16.644568 10124 ts_manager.cc:194] Registered new tserver with Master: 735f8a385a4e4ff99d4cdac0b971b01b (127.1.210.2:42633)
I20260924 11:48:16.649194 10124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:59835
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:48:16.683629 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:45337
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:16.997171 10452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:16.997426 10452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:16.997519 10452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:17.004621 10452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:17.004719 10452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:17.004828 10452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:17.014354 10452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:17.015519 10452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:17.016700 10452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:17.029259 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:17.029330 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:17.029350 10460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:17.030011 10452 server_base.cc:1034] running on GCE node
I20260924 11:48:17.030532 10452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:17.031342 10452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:17.032529 10452 hybrid_clock.cc:648] HybridClock initialized: now 1790250497032486 us; error 44 us; skew 500 ppm
I20260924 11:48:17.032785 10452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:17 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250497, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:17.035773 10452 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:48:17.037010 10452 webserver.cc:460] Webserver started at http://127.1.210.3:43519/ using document root <none> and password file <none>
I20260924 11:48:17.037678 10452 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:17.037771 10452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:17.038010 10452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:17.040624 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "a7f59687b4e040f4aafdd9b486ddf6f6"
format_stamp: "Formatted at 2026-09-24 11:48:17 on dist-test-slave-dd32"
server_key: "3668cf6ddc8a5d0bd1efa1891b1f7713"
server_key_iv: "3b613aa546cafa5daad51c4c1aae5413"
server_key_version: "encryptionkey@0"
I20260924 11:48:17.041380 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "a7f59687b4e040f4aafdd9b486ddf6f6"
format_stamp: "Formatted at 2026-09-24 11:48:17 on dist-test-slave-dd32"
server_key: "3668cf6ddc8a5d0bd1efa1891b1f7713"
server_key_iv: "3b613aa546cafa5daad51c4c1aae5413"
server_key_version: "encryptionkey@0"
I20260924 11:48:17.046136 10452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:48:17.049656 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:17.051018 10452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:48:17.051141 10452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "a7f59687b4e040f4aafdd9b486ddf6f6"
format_stamp: "Formatted at 2026-09-24 11:48:17 on dist-test-slave-dd32"
server_key: "3668cf6ddc8a5d0bd1efa1891b1f7713"
server_key_iv: "3b613aa546cafa5daad51c4c1aae5413"
server_key_version: "encryptionkey@0"
I20260924 11:48:17.051291 10452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:17.063699 10452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:17.067448 10452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:17.067708 10452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:17.068637 10452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:17.070006 10452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:17.070075 10452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:17.070212 10452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:17.070288 10452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:17.101181 10452 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:39565
I20260924 11:48:17.101227 10579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:39565 every 8 connection(s)
I20260924 11:48:17.102560 10452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:17.112392 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10452
I20260924 11:48:17.112560 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:48:17.113125 1864 external_mini_cluster.cc:1428] Setting key 1c42e547f6a07721fbc58ba331355d39
Sep 24 11:48:17 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250497, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:17.122059 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:17.104923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:50057 (local address 127.1.210.62:45337)
0924 11:48:17.105149 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:17.105153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:17.106255 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 11:48:17.108055 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:17.108061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:17.108162 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:17.108280 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:17.109770 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.110515 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.112020 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.112429 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.112570 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:17.116702 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:17.116707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:17.116710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:17.116740 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:17.119368 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.119967 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.119971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.119973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.120090 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.120426 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.120429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.120431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.120670 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:17.120767 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:17.121433 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 11:48:17.121676 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:48:17.122838 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:17.105442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:50057)
0924 11:48:17.106107 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:17.106143 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:17.107785 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:17.108522 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:17.108529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:17.108993 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:17.109609 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:17.109625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.110666 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.110676 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:17.111881 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:17.111892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.112717 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.112725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:17.113026 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:17.113726 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:17.113756 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:17.116534 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:17.119476 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:17.119482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:17.119493 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:17.119867 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:17.120176 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:17.120179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:17.120181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:17.120334 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:17.120762 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:17.120767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:17.121059 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 11:48:17.121334 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 11:48:17.121690 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":255,"threads_started":1}
I20260924 11:48:17.124358 10580 heartbeater.cc:347] Connected to a master server at 127.1.210.62:45337
I20260924 11:48:17.124646 10580 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:17.125371 10580 heartbeater.cc:511] Master 127.1.210.62:45337 requested a full tablet report, sending...
I20260924 11:48:17.126771 10124 ts_manager.cc:194] Registered new tserver with Master: a7f59687b4e040f4aafdd9b486ddf6f6 (127.1.210.3:39565)
I20260924 11:48:17.127869 10124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:50057
I20260924 11:48:17.128469 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.1@KRBTEST.COM" created.
I20260924 11:48:17.157176 10313 heartbeater.cc:503] Master 127.1.210.62:45337 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab.
Entry for principal oryx/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab.
WARNING: no policy specified for oryx/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab.
Entry for principal oryx/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab.
WARNING: no policy specified for oryx/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab.
Entry for principal oryx/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab.
I20260924 11:48:17.248986 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10185
Sep 24 11:48:17 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:48:17.268472 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:17.249988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.1.210.1:43723)
0924 11:48:17.250605 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:17.250613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:17.250641 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:48:17.250826 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:17.250833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:17.251084 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:17.251221 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:17.252597 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.253676 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.254978 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.255575 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.255682 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:17.263152 (+ 7470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:17.263163 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:17.263177 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:17.263208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:17.266035 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.266613 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.266619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.266622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.266750 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.267053 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.267056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.267058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.267432 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:17.267652 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:17.267893 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 11:48:17.268091 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":301,"threads_started":1}
I20260924 11:48:17.269891 10308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:17.351755 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10185
I20260924 11:48:17.368439 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10319
Sep 24 11:48:17 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:48:17.383391 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:17.369303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.1.210.2:42633)
0924 11:48:17.369759 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:17.369766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:17.369790 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:48:17.369848 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:17.369855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:17.370053 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:17.370177 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:17.371417 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.372409 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.373648 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.374215 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.374326 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:17.378198 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:17.378208 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:17.378221 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:17.378249 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:17.381037 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.381660 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.381666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.381669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.381784 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.382090 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.382093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.382095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.382401 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:17.382608 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:17.382825 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 11:48:17.383006 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":228,"threads_started":1}
I20260924 11:48:17.384563 10442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:17.466660 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10319
I20260924 11:48:17.482231 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10452
Sep 24 11:48:17 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:48:17.497802 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:17.483127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.1.210.3:39565)
0924 11:48:17.483705 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:17.483712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:17.483739 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:48:17.483796 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:17.483802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:17.484026 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:17.484188 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:17.485509 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.486517 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.487807 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:17.488436 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:17.488576 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:17.492601 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:17.492611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:17.492624 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:17.492652 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:17.495185 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.495808 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.495816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.495818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.495932 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:17.496335 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:17.496340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:17.496343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:17.496644 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:17.496955 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:17.497244 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:48:17.497492 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":301,"threads_started":1}
I20260924 11:48:17.499007 10575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:17.580464 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10452
I20260924 11:48:17.595714 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10092
I20260924 11:48:17.596856 10154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:17.697012 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10092
I20260924 11:48:17.717206 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:45337
--webserver_interface=127.1.210.62
--webserver_port=41635
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:18.030195 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:18.030453 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:18.030627 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:18.037644 10597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:18.037763 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:18.037864 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:18.037957 10597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:18.038053 10597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:18.047555 10597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=41635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:18.048779 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:18.050032 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:18.061960 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:18.061878 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:18.061872 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:18.062722 10597 server_base.cc:1034] running on GCE node
I20260924 11:48:18.063441 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:18.064736 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:18.065958 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790250498065926 us; error 45 us; skew 500 ppm
I20260924 11:48:18.066275 10597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250498, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:18.069247 10597 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:48:18.070537 10597 webserver.cc:460] Webserver started at http://127.1.210.62:41635/ using document root <none> and password file <none>
I20260924 11:48:18.071192 10597 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:18.071342 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:18.076366 10597 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:48:18.078980 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:18.080147 10597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:18.080324 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1"
format_stamp: "Formatted at 2026-09-24 11:48:15 on dist-test-slave-dd32"
server_key: "e641205154e72c2eb321cce3cb6e6daf"
server_key_iv: "de9711d35b73a02b821777df9b0370bb"
server_key_version: "encryptionkey@0"
I20260924 11:48:18.081306 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:18.095257 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:18.098950 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:18.099205 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:18.116685 10597 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:45337
I20260924 11:48:18.116704 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:45337 every 8 connection(s)
I20260924 11:48:18.118108 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:48:18.124447 10664 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:48:18.125554 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10597
I20260924 11:48:18.126279 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:43723
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:45337
--webserver_port=33089
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:48:18.127565 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Bootstrap starting.
I20260924 11:48:18.152129 10664 log.cc:824] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:18.162873 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:48:18.163447 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Bootstrap complete.
I20260924 11:48:18.172744 10664 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:18.173683 10664 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e3b48fc2e04bd1bca574b16f58f2d1, State: Initialized, Role: FOLLOWER
I20260924 11:48:18.174340 10664 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:18.174580 10664 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:48:18.174700 10664 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:48:18.174887 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:48:18.178499 10664 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:18.178963 10664 leader_election.cc:302] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e3b48fc2e04bd1bca574b16f58f2d1; no voters:
I20260924 11:48:18.179518 10664 leader_election.cc:288] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:48:18.179708 10668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:48:18.180956 10668 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [term 2 LEADER]: Becoming Leader. State: Replica: a9e3b48fc2e04bd1bca574b16f58f2d1, State: Running, Role: LEADER
I20260924 11:48:18.181660 10664 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:18.181636 10668 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } }
I20260924 11:48:18.185312 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } } }
I20260924 11:48:18.185643 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:18.184839 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e3b48fc2e04bd1bca574b16f58f2d1. Latest consensus state: current_term: 2 leader_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e3b48fc2e04bd1bca574b16f58f2d1" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 45337 } } }
I20260924 11:48:18.186267 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:18.194334 10676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:18.200114 10676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:18.201697 10676 catalog_manager.cc:1259] Loaded cluster ID: 8988f2470ce1448390541c64a0f8f0d2
I20260924 11:48:18.201790 10676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:18.205716 10676 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:18.207021 10676 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9e3b48fc2e04bd1bca574b16f58f2d1: Loaded TSK: 0
I20260924 11:48:18.208420 10676 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:48:18.478984 10666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:18.479290 10666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:18.479480 10666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:18.486881 10666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:18.487022 10666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:18.487187 10666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:18.497044 10666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.1.keytab
--principal=oryx/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:43723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=33089
--tserver_master_addrs=127.1.210.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:18.498492 10666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:18.499828 10666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:18.514694 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:18.514660 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:18.514761 10694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:18.516029 10666 server_base.cc:1034] running on GCE node
I20260924 11:48:18.516600 10666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:18.517622 10666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:18.518832 10666 hybrid_clock.cc:648] HybridClock initialized: now 1790250498518769 us; error 59 us; skew 500 ppm
I20260924 11:48:18.519135 10666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250498, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:18.522259 10666 init.cc:377] Logged in from keytab as oryx/127.1.210.1@KRBTEST.COM (short username oryx)
I20260924 11:48:18.523792 10666 webserver.cc:460] Webserver started at http://127.1.210.1:33089/ using document root <none> and password file <none>
I20260924 11:48:18.524462 10666 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:18.524597 10666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:18.529914 10666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:48:18.532631 10701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:18.534045 10666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:48:18.534206 10666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "929b4dea8f32472ead3b0ea63ba70c9b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "edaf4fb962efb4e1b4781465e74a9468"
server_key_iv: "3576b5713e1e92011ee7b54bc8d7b022"
server_key_version: "encryptionkey@0"
I20260924 11:48:18.535101 10666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:18.547988 10666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:18.552178 10666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:18.552446 10666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:18.553539 10666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:18.554950 10666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:18.555042 10666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:18.555183 10666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:18.555298 10666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:18.587347 10666 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:43723
I20260924 11:48:18.587417 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:43723 every 8 connection(s)
I20260924 11:48:18.588759 10666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:18.591607 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10666
I20260924 11:48:18.592365 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:42633
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:45337
--webserver_port=34417
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:48:18.608372 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:18.591453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:49455 (local address 127.1.210.62:45337)
0924 11:48:18.592117 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:18.592134 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:18.592667 (+ 533us) server_negotiation.cc:407] Connection header received
0924 11:48:18.594846 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:18.594859 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:18.595162 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:18.595542 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:18.596966 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.598925 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.600416 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.601043 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.601269 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:18.607545 (+ 6276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:49455: BlockingRecv error: recv got EOF from 127.1.210.1:49455 (error 108)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":255,"threads_started":1}
I20260924 11:48:18.608564 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:18.591692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:49455)
0924 11:48:18.592475 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:18.592510 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:18.594318 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:18.595673 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:18.595686 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:18.596265 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:18.596769 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:18.596786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.599078 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.599091 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:18.600270 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:18.600281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.601299 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.601306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:18.601558 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:18.602279 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:18.602313 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:18.607255 (+ 4942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":330,"threads_started":1}
W20260924 11:48:18.609282 10814 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:45337 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.210.62:45337: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:18.618631 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:18.609784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:55325)
0924 11:48:18.610004 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:18.610025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:18.610133 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:18.610434 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:18.610442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:18.610862 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:18.611072 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:18.611085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.612013 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.612021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:18.613778 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:18.613796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.614453 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.614462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:18.614685 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:18.615222 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:18.615251 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:18.618379 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 11:48:18.618671 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:18.609937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:55325 (local address 127.1.210.62:45337)
0924 11:48:18.610063 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:18.610067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:18.610082 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:48:18.610240 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:18.610243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:18.610291 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:18.610371 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:18.611214 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.611875 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.613931 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.614300 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.614395 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:18.618521 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:55325: BlockingRecv error: recv got EOF from 127.1.210.1:55325 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:18.627313 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:18.619183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:38793)
0924 11:48:18.619359 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:18.619376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:18.619459 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:18.619793 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:18.619798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:18.620138 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:18.620303 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:18.620312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.621267 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.621272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:18.622789 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:18.622808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.623428 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.623434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:18.623624 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:18.624113 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:18.624134 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:18.627036 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 11:48:18.627555 10814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:48:18.627597 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:18.619409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:38793 (local address 127.1.210.62:45337)
0924 11:48:18.619571 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:18.619574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:18.619588 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:48:18.619621 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:18.619625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:18.619679 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:18.619751 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:18.620433 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.621146 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.622921 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:18.623290 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:18.623383 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:18.627178 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:38793: BlockingRecv error: recv got EOF from 127.1.210.1:38793 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 11:48:18.919476 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:18.919728 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:18.919842 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:18.927141 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:18.927275 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:18.927486 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:18.941016 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.2.keytab
--principal=oryx/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:42633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=34417
--tserver_master_addrs=127.1.210.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:18.942364 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:18.943699 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:18.956830 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:18.956771 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:18.956875 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:18.957671 10818 server_base.cc:1034] running on GCE node
I20260924 11:48:18.958315 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:18.959194 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:18.960384 10818 hybrid_clock.cc:648] HybridClock initialized: now 1790250498960333 us; error 56 us; skew 500 ppm
I20260924 11:48:18.960685 10818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:18 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250498, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:18.963845 10818 init.cc:377] Logged in from keytab as oryx/127.1.210.2@KRBTEST.COM (short username oryx)
I20260924 11:48:18.965189 10818 webserver.cc:460] Webserver started at http://127.1.210.2:34417/ using document root <none> and password file <none>
I20260924 11:48:18.965803 10818 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:18.965916 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:18.970839 10818 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:48:18.973477 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:18.974747 10818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:18.974921 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "735f8a385a4e4ff99d4cdac0b971b01b"
format_stamp: "Formatted at 2026-09-24 11:48:16 on dist-test-slave-dd32"
server_key: "3d2a6f02ffa61a7cce67ef7098cf6a6f"
server_key_iv: "b5c9ed719fc05c0d96ee6ab888a69725"
server_key_version: "encryptionkey@0"
I20260924 11:48:18.975832 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:18.996165 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:19.000344 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:19.000603 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:19.001554 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:19.002810 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:19.002938 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:19.003065 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:19.003154 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:19.035518 10818 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:42633
I20260924 11:48:19.035602 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:42633 every 8 connection(s)
I20260924 11:48:19.036902 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:19.044546 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10818
I20260924 11:48:19.045331 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:39565
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:45337
--webserver_port=43519
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:48:19.053369 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.039324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:47141 (local address 127.1.210.62:45337)
0924 11:48:19.039509 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.039515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.040694 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 11:48:19.042429 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.042433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.042493 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.042579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.043957 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.044631 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.047228 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.047638 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.047735 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.053118 (+ 5383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:47141: BlockingRecv error: recv got EOF from 127.1.210.2:47141 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:48:19.054052 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.039865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:47141)
0924 11:48:19.040537 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.040580 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.042190 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.042775 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.042782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.043259 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.043812 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.043828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.044844 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.044850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.047073 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.047089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.047875 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.047883 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.048190 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.048904 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.048942 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.052925 (+ 3983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":263,"threads_started":1}
W20260924 11:48:19.054675 10946 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:45337 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.210.62:45337: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:19.063117 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.055261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:44019)
0924 11:48:19.055439 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.055460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.055538 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.055859 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.055863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.056133 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.056255 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.056262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.057730 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.057734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.058597 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.058607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.059590 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.059594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.059708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.060284 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.060300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.062921 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:48:19.063972 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.055367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:44019 (local address 127.1.210.62:45337)
0924 11:48:19.055565 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.055573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.055592 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:48:19.055644 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.055651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.055727 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.055826 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.056400 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.057591 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.058736 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.059463 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.059597 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.063642 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:44019: BlockingRecv error: recv got EOF from 127.1.210.2:44019 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:19.072263 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.063888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:47265)
0924 11:48:19.064124 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.064141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.064235 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.064641 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.064649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.065007 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.065198 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.065206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.066662 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.066666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.067534 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.067544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.068565 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.068569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.068703 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.069283 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.069299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.071982 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260924 11:48:19.072628 10946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:48:19.073082 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.064053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:47265 (local address 127.1.210.62:45337)
0924 11:48:19.064301 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.064308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.064329 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:48:19.064395 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.064402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.064480 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.064593 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.065333 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.066538 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.067678 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.068433 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.068582 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.072217 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:47265: BlockingRecv error: recv got EOF from 127.1.210.2:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":756,"threads_started":1}
W20260924 11:48:19.371675 10949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:19.372013 10949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:19.372200 10949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:19.379568 10949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:19.379729 10949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:19.379897 10949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:19.389909 10949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/krb5kdc/oryx_127.1.210.3.keytab
--principal=oryx/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:39565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=43519
--tserver_master_addrs=127.1.210.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:19.391243 10949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:19.392711 10949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:19.406596 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:19.406682 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:19.406651 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:19.407547 10949 server_base.cc:1034] running on GCE node
I20260924 11:48:19.408172 10949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:19.409035 10949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:19.410274 10949 hybrid_clock.cc:648] HybridClock initialized: now 1790250499410233 us; error 60 us; skew 500 ppm
I20260924 11:48:19.410553 10949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250499, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:19.413647 10949 init.cc:377] Logged in from keytab as oryx/127.1.210.3@KRBTEST.COM (short username oryx)
I20260924 11:48:19.414927 10949 webserver.cc:460] Webserver started at http://127.1.210.3:43519/ using document root <none> and password file <none>
I20260924 11:48:19.415573 10949 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:19.415719 10949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:19.420754 10949 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:48:19.423305 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:19.424650 10949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:48:19.424871 10949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "a7f59687b4e040f4aafdd9b486ddf6f6"
format_stamp: "Formatted at 2026-09-24 11:48:17 on dist-test-slave-dd32"
server_key: "3668cf6ddc8a5d0bd1efa1891b1f7713"
server_key_iv: "3b613aa546cafa5daad51c4c1aae5413"
server_key_version: "encryptionkey@0"
I20260924 11:48:19.425959 10949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:19.444588 10949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:19.448369 10949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:19.448653 10949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:19.449667 10949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:19.451007 10949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:19.451136 10949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:19.451292 10949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:19.451390 10949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:19.486826 10949 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:39565
I20260924 11:48:19.486876 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:39565 every 8 connection(s)
I20260924 11:48:19.488371 10949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:19.488914 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 10949
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:19.504581 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.489919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.1.210.62:45337)
0924 11:48:19.490134 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.490140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.490622 (+ 482us) server_negotiation.cc:407] Connection header received
0924 11:48:19.490825 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.490836 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.490904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.491006 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.492600 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.493583 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.494805 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.495282 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.495412 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.497977 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:19.497985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:19.497998 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:19.498031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:19.501256 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:19.501965 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:19.501971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:19.501977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:19.502136 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:19.502569 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:19.502577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:19.502583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:19.503011 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:19.503349 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:19.503826 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 11:48:19.504169 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 11:48:19.504904 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.491227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:56735 (local address 127.1.210.62:45337)
0924 11:48:19.491410 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.491416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.492817 (+ 1401us) server_negotiation.cc:407] Connection header received
0924 11:48:19.494752 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.494757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.494836 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.494982 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.496472 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.497316 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.498919 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.499460 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.499602 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.504658 (+ 5056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:56735: BlockingRecv error: recv got EOF from 127.1.210.3:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:48:19.505447 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.491768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:56735)
0924 11:48:19.492644 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.492706 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.494484 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.495238 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.495248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.495753 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.496294 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.496310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.497473 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.497479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.498765 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.498778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.499699 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.499708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.499964 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.500681 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.500704 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.504347 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":372,"threads_started":1}
W20260924 11:48:19.506098 11078 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:45337 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.210.62:45337: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:19.514312 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.506637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:55175)
0924 11:48:19.506918 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.506938 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.507053 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.507872 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.507876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.508146 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.508321 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.508333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.509443 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.509447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.510290 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.510299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.510921 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.510924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.511033 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.511548 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.511563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.514131 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260924 11:48:19.514554 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.506759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:55175 (local address 127.1.210.62:45337)
0924 11:48:19.506928 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.506937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.506997 (+ 60us) server_negotiation.cc:407] Connection header received
0924 11:48:19.507530 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.507534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.507584 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.507673 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.508486 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.509305 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.510410 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.510807 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.510903 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.514298 (+ 3395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:55175: BlockingRecv error: recv got EOF from 127.1.210.3:55175 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:19.521752 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.514896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:52761)
0924 11:48:19.515055 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.515069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.515138 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.515399 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.515402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.515638 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.515764 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.515770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.516659 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.516662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.517603 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.517613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.518221 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.518225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.518368 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.518920 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.518936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.521552 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:48:19.522044 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.515002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:52761 (local address 127.1.210.62:45337)
0924 11:48:19.515192 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.515196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.515209 (+ 13us) server_negotiation.cc:407] Connection header received
0924 11:48:19.515245 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.515248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.515292 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.515370 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.515883 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.516547 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.517713 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.518111 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.518199 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.521703 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:52761: BlockingRecv error: recv got EOF from 127.1.210.3:52761 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 11:48:19.522074 11078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:19 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:19.636314 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:19.628087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:47033)
0924 11:48:19.628513 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:19.628527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:19.628694 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:19.629023 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:19.629027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:19.629578 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:19.629791 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.629799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.630792 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.630799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.631709 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:19.631719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.632471 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.632477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:19.632648 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.633272 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:19.633289 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:19.636006 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
W20260924 11:48:19.636405 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:19.628114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:47033 (local address 127.1.210.62:45337)
0924 11:48:19.628359 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:19.628365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:19.628629 (+ 264us) server_negotiation.cc:407] Connection header received
0924 11:48:19.628819 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:19.628824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:19.628899 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:19.628997 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:19.629928 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.630672 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.631846 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:19.632243 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:19.632429 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:19.636176 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:47033: BlockingRecv error: recv got EOF from 127.1.210.1:47033 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:48:20 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:20 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:20.081708 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:20.073219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:57695)
0924 11:48:20.073701 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:20.073715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:20.073876 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:20.074207 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:20.074212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:20.074661 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:20.074840 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:20.074848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.075907 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.075914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:20.076850 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:20.076870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.077667 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.077673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:20.077837 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:20.078467 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:20.078484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:20.081385 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
W20260924 11:48:20.081817 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:20.073278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57695 (local address 127.1.210.62:45337)
0924 11:48:20.073495 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:20.073503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:20.073796 (+ 293us) server_negotiation.cc:407] Connection header received
0924 11:48:20.073997 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:20.074002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:20.074072 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:20.074176 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:20.074991 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.075739 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.077000 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.077487 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.077613 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:20.081578 (+ 3965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:57695: BlockingRecv error: recv got EOF from 127.1.210.2:57695 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 11:48:20 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:20 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:20.531306 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:20.522681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:39983)
0924 11:48:20.523194 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:20.523208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:20.523378 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:20.523703 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:20.523707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:20.524165 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:20.524339 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:20.524347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.525417 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.525424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:20.526381 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:20.526392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.527115 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.527123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:20.527281 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:20.528207 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:20.528232 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:20.531089 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":273,"threads_started":1}
W20260924 11:48:20.531477 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:20.522705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:39983 (local address 127.1.210.62:45337)
0924 11:48:20.522920 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:20.522928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:20.523309 (+ 381us) server_negotiation.cc:407] Connection header received
0924 11:48:20.523502 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:20.523509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:20.523574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:20.523674 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:20.524491 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.525280 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.526524 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.526928 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.527059 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:20.531267 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:39983: BlockingRecv error: recv got EOF from 127.1.210.3:39983 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 11:48:20 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:20 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:20.644976 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:20.637401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:40715)
0924 11:48:20.637838 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:20.637852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:20.637974 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:20.638323 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:20.638328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:20.638683 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:20.638822 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:20.638830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.639838 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.639841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:20.640709 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:20.640720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.641355 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.641359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:20.641467 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:20.642021 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:20.642037 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:20.644637 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":249,"threads_started":1}
W20260924 11:48:20.645138 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:20.637427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:40715 (local address 127.1.210.62:45337)
0924 11:48:20.637648 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:20.637658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:20.637940 (+ 282us) server_negotiation.cc:407] Connection header received
0924 11:48:20.638077 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:20.638080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:20.638148 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:20.638223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:20.638971 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.639731 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.640825 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:20.641235 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:20.641323 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:20.644835 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:40715: BlockingRecv error: recv got EOF from 127.1.210.1:40715 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 11:48:21 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:21 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:21.090619 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:21.082651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:53503)
0924 11:48:21.083167 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:21.083182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:21.083333 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:21.083652 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:21.083657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:21.084073 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:21.084211 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:21.084225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.085367 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.085372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:21.086338 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:21.086349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.087045 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.087050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:21.087217 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:21.087728 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:21.087744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:21.090381 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
W20260924 11:48:21.090778 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:21.082690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:53503 (local address 127.1.210.62:45337)
0924 11:48:21.082940 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:21.082947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:21.083267 (+ 320us) server_negotiation.cc:407] Connection header received
0924 11:48:21.083454 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:21.083459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:21.083526 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:21.083624 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:21.084359 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.085238 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.086476 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.086895 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.087029 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:21.090548 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:53503: BlockingRecv error: recv got EOF from 127.1.210.2:53503 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 11:48:21 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:21 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:21.540591 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:21.532113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:48413)
0924 11:48:21.532627 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:21.532642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:21.532793 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:21.533158 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:21.533164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:21.533568 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:21.533716 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:21.533723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.534743 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.534750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:21.535653 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:21.535664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.536328 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.536333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:21.536499 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:21.537193 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:21.537210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:21.540238 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
W20260924 11:48:21.540640 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:21.532304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:48413 (local address 127.1.210.62:45337)
0924 11:48:21.532512 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:21.532518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:21.532728 (+ 210us) server_negotiation.cc:407] Connection header received
0924 11:48:21.532925 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:21.532930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:21.532997 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:21.533114 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:21.533848 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.534611 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.535793 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.536178 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.536317 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:21.540425 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:48413: BlockingRecv error: recv got EOF from 127.1.210.3:48413 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 11:48:21 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:21 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:21.654111 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:21.645853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:44563)
0924 11:48:21.646286 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:21.646301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:21.646424 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:21.646715 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:21.646719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:21.647045 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:21.647196 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:21.647203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.648212 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.648215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:21.649510 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:21.649524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.650346 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.650350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:21.650475 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:21.651014 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:21.651031 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:21.653787 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":243,"threads_started":1}
W20260924 11:48:21.654194 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:21.645949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:44563 (local address 127.1.210.62:45337)
0924 11:48:21.646182 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:21.646188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:21.646381 (+ 193us) server_negotiation.cc:407] Connection header received
0924 11:48:21.646536 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:21.646539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:21.646597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:21.646672 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:21.647306 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.648079 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.649663 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:21.650224 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:21.650321 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:21.653994 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:44563: BlockingRecv error: recv got EOF from 127.1.210.1:44563 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 11:48:22 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:22 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:22.099330 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:22.091545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:59075)
0924 11:48:22.092042 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:22.092058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:22.092249 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:22.092574 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:22.092578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:22.092910 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:22.093137 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:22.093154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.094133 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.094137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:22.095021 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:22.095031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.095685 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.095689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:22.095798 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:22.096467 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:22.096484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:22.099054 (+ 2570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":279,"threads_started":1}
W20260924 11:48:22.099467 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:22.091608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:59075 (local address 127.1.210.62:45337)
0924 11:48:22.091786 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:22.091794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:22.092139 (+ 345us) server_negotiation.cc:407] Connection header received
0924 11:48:22.092349 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:22.092352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:22.092426 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:22.092520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:22.093299 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.094018 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.095145 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.095574 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.095661 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:22.099231 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:59075: BlockingRecv error: recv got EOF from 127.1.210.2:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 11:48:22 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:22 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:22.557699 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:22.541587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:58233)
0924 11:48:22.542079 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:22.542095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:22.542252 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:22.542587 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:22.542592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:22.542945 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:22.543101 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:22.543110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.544134 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.544142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:22.545104 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:22.545114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.545842 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.545847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:22.546025 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:22.554752 (+ 8727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:22.554770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:22.557463 (+ 2693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":256,"threads_started":1}
W20260924 11:48:22.557861 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:22.541653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:58233 (local address 127.1.210.62:45337)
0924 11:48:22.541871 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:22.541881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:22.542183 (+ 302us) server_negotiation.cc:407] Connection header received
0924 11:48:22.542373 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:22.542378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:22.542453 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:22.542553 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:22.543250 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.543994 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.545272 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.545701 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.545829 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:22.557643 (+ 11814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:58233: BlockingRecv error: recv got EOF from 127.1.210.3:58233 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 11:48:22 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:22 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:22.662195 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:22.654887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:35855)
0924 11:48:22.655332 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:22.655348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:22.655468 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:22.655755 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:22.655758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:22.656086 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:22.656225 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:22.656233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.657194 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.657198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:22.658099 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:22.658109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.658719 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.658723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:22.658841 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:22.659377 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:22.659393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:22.661982 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":241,"threads_started":1}
W20260924 11:48:22.662369 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:22.654983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:35855 (local address 127.1.210.62:45337)
0924 11:48:22.655245 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:22.655251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:22.655422 (+ 171us) server_negotiation.cc:407] Connection header received
0924 11:48:22.655565 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:22.655569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:22.655627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:22.655702 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:22.656338 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.657095 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.658210 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:22.658608 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:22.658693 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:22.662148 (+ 3455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:35855: BlockingRecv error: recv got EOF from 127.1.210.1:35855 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 24 11:48:23 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:23 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:23.107580 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:23.100123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:51579)
0924 11:48:23.100624 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:23.100641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:23.100750 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:23.101033 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:23.101036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:23.101392 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:23.101532 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:23.101540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.102485 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.102488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:23.103443 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:23.103454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.104084 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.104087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:23.104221 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:23.104773 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:23.104789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:23.107362 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":273,"threads_started":1}
W20260924 11:48:23.107810 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:23.100208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:51579 (local address 127.1.210.62:45337)
0924 11:48:23.100366 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:23.100370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:23.100714 (+ 344us) server_negotiation.cc:407] Connection header received
0924 11:48:23.100844 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:23.100847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:23.100913 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:23.100989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:23.101645 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.102378 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.103561 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.103969 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.104056 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:23.107578 (+ 3522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:51579: BlockingRecv error: recv got EOF from 127.1.210.2:51579 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 11:48:23 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:23 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:23.566777 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:23.558633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:57429)
0924 11:48:23.559165 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:23.559181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:23.559338 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:23.559625 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:23.559628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:23.559961 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:23.560105 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:23.560113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.561089 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.561116 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:23.562080 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:23.562092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.562707 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.562711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:23.562845 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:23.563417 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:23.563434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:23.566498 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
W20260924 11:48:23.566983 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:23.558721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:57429 (local address 127.1.210.62:45337)
0924 11:48:23.558942 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:23.558950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:23.559253 (+ 303us) server_negotiation.cc:407] Connection header received
0924 11:48:23.559439 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:23.559443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:23.559499 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:23.559581 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:23.560238 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.560938 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.562200 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.562590 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.562681 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:23.566711 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:57429: BlockingRecv error: recv got EOF from 127.1.210.3:57429 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 11:48:23 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:23 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:23.670713 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:23.663098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:46393)
0924 11:48:23.663569 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:23.663585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:23.663693 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:23.663975 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:23.663978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:23.664360 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:23.664508 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:23.664516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.665514 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.665518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:23.666559 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:23.666570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.667160 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.667164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:23.667276 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:23.667807 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:23.667823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:23.670487 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":196,"threads_started":1}
W20260924 11:48:23.671070 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:23.663388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:46393 (local address 127.1.210.62:45337)
0924 11:48:23.663586 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:23.663590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:23.663656 (+ 66us) server_negotiation.cc:407] Connection header received
0924 11:48:23.663790 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:23.663794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:23.663848 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:23.663927 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:23.664628 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.665400 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.666678 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:23.667031 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:23.667128 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:23.670732 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:46393: BlockingRecv error: recv got EOF from 127.1.210.1:46393 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 11:48:24 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:24 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:24.116151 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:24.108473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:60123)
0924 11:48:24.109025 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:24.109040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:24.109164 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:24.109415 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:24.109418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:24.109782 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:24.109954 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:24.109963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.111100 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.111104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:24.111990 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:24.112001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.112628 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.112632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:24.112741 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:24.113323 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:24.113339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:24.115898 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":335,"threads_started":1}
W20260924 11:48:24.116242 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:24.108660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:60123 (local address 127.1.210.62:45337)
0924 11:48:24.108810 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:24.108814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:24.109137 (+ 323us) server_negotiation.cc:407] Connection header received
0924 11:48:24.109243 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:24.109246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:24.109301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:24.109374 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:24.110073 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.110973 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.112109 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.112515 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.112602 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:24.116073 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:60123: BlockingRecv error: recv got EOF from 127.1.210.2:60123 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 11:48:24 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:24 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:24.576038 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:24.567747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:56079)
0924 11:48:24.568221 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:24.568236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:24.568337 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:24.568592 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:24.568595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:24.568989 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:24.569158 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:24.569166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.570360 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.570363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:24.571251 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:24.571261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.571892 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.571896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:24.572022 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:24.572659 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:24.572675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:24.575664 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
W20260924 11:48:24.576200 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:24.567788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:56079 (local address 127.1.210.62:45337)
0924 11:48:24.567986 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:24.567995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:24.568310 (+ 315us) server_negotiation.cc:407] Connection header received
0924 11:48:24.568414 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:24.568417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:24.568482 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:24.568555 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:24.569271 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.570166 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.571386 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.571770 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.571863 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:24.575930 (+ 4067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:56079: BlockingRecv error: recv got EOF from 127.1.210.3:56079 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 11:48:24 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:24 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:24.679075 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:24.671567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:50727)
0924 11:48:24.672078 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:24.672094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:24.672203 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:24.672496 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:24.672499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:24.672848 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:24.672996 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:24.673004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.673959 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.673963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:24.674835 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:24.674845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.675491 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.675495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:24.675607 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:24.676155 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:24.676189 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:24.678784 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":287,"threads_started":1}
W20260924 11:48:24.679332 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:24.671605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:50727 (local address 127.1.210.62:45337)
0924 11:48:24.671767 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:24.671772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:24.672168 (+ 396us) server_negotiation.cc:407] Connection header received
0924 11:48:24.672315 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:24.672318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:24.672375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:24.672451 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:24.673169 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.673837 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.674961 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:24.675356 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:24.675447 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:24.679040 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:50727: BlockingRecv error: recv got EOF from 127.1.210.1:50727 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:48:25 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:25 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:25.126730 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:25.117267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:42527)
0924 11:48:25.117977 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:25.117995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:25.118152 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:25.118492 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:25.118497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:25.118956 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:25.119127 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:25.119138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.120312 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.120317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:25.121754 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:25.121769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.122514 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.122520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:25.122692 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:25.123270 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:25.123291 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:25.126406 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":450,"threads_started":1}
W20260924 11:48:25.126806 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:25.117391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:42527 (local address 127.1.210.62:45337)
0924 11:48:25.117573 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:25.117579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:25.118063 (+ 484us) server_negotiation.cc:407] Connection header received
0924 11:48:25.118288 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:25.118300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:25.118360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:25.118438 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:25.119255 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.120187 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.121883 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.122355 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.122447 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:25.126591 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:42527: BlockingRecv error: recv got EOF from 127.1.210.2:42527 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:48:25 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:25 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:25.585011 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:25.576934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:40503)
0924 11:48:25.577464 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:25.577479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:25.577628 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:25.577952 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:25.577958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:25.578390 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:25.578532 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:25.578540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.579656 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.579663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:25.580563 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:25.580574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.581324 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.581330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:25.581513 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:25.582089 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:25.582105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:25.584797 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":300,"threads_started":1}
W20260924 11:48:25.585251 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:25.576997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:40503 (local address 127.1.210.62:45337)
0924 11:48:25.577250 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:25.577257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:25.577561 (+ 304us) server_negotiation.cc:407] Connection header received
0924 11:48:25.577754 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:25.577763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:25.577817 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:25.577913 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:25.578662 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.579527 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.580705 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.581174 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.581306 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:25.584957 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:40503: BlockingRecv error: recv got EOF from 127.1.210.3:40503 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 11:48:25 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:25 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:25.688594 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:25.680025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:52003)
0924 11:48:25.680638 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:25.680653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:25.680803 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:25.681269 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:25.681276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:25.681832 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:25.682007 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:25.682030 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.683073 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.683079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:25.683990 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:25.684001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.684783 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.684789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:25.684951 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:25.685665 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:25.685683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:25.688339 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":281,"threads_started":1}
W20260924 11:48:25.688709 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:25.680120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:52003 (local address 127.1.210.62:45337)
0924 11:48:25.680279 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:25.680283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:25.680740 (+ 457us) server_negotiation.cc:407] Connection header received
0924 11:48:25.680992 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:25.681000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:25.681110 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:25.681234 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:25.682157 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.682931 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.684121 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:25.684635 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:25.684767 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:25.688501 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:52003: BlockingRecv error: recv got EOF from 127.1.210.1:52003 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 11:48:26 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:26 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:26.137630 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:26.127747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:33165)
0924 11:48:26.128307 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:26.128323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:26.128430 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:26.128713 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:26.128716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:26.129149 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:26.129312 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:26.129321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.130323 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.130327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:26.131251 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:26.131262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.131898 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.131902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:26.132023 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:26.132540 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:26.132557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:26.137377 (+ 4820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":272,"threads_started":1}
W20260924 11:48:26.137851 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:26.127795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:33165 (local address 127.1.210.62:45337)
0924 11:48:26.128050 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:26.128059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:26.128399 (+ 340us) server_negotiation.cc:407] Connection header received
0924 11:48:26.128524 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:26.128527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:26.128584 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:26.128661 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:26.129429 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.130195 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.131364 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.131764 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.131852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:26.137599 (+ 5747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:33165: BlockingRecv error: recv got EOF from 127.1.210.2:33165 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 24 11:48:26 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:26 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:26.594205 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:26.585930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:33145)
0924 11:48:26.586582 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:26.586598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:26.586754 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:26.587091 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:26.587095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:26.587517 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:26.587726 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:26.587738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.588734 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.588741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:26.589746 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:26.589757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.590441 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.590447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:26.590620 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:26.591171 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:26.591188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:26.593956 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":308,"threads_started":1}
W20260924 11:48:26.594326 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:26.585963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:33145 (local address 127.1.210.62:45337)
0924 11:48:26.586180 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:26.586189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:26.586685 (+ 496us) server_negotiation.cc:407] Connection header received
0924 11:48:26.586888 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:26.586896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:26.586962 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:26.587063 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:26.587873 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.588598 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.589884 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.590290 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.590425 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:26.594121 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:33145: BlockingRecv error: recv got EOF from 127.1.210.3:33145 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 11:48:26 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:26 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:26.698175 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:26.689455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:57979)
0924 11:48:26.689990 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:26.690005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:26.690116 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:26.690403 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:26.690406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:26.690799 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:26.690944 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:26.690952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.692149 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.692158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:26.693563 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:26.693575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.694238 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.694242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:26.694349 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:26.694923 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:26.694940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:26.697823 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":300,"threads_started":1}
W20260924 11:48:26.698266 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:26.689625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:57979 (local address 127.1.210.62:45337)
0924 11:48:26.689779 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:26.689783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:26.690075 (+ 292us) server_negotiation.cc:407] Connection header received
0924 11:48:26.690212 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:26.690215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:26.690284 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:26.690363 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:26.691059 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.691979 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.693679 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:26.694094 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:26.694189 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:26.698030 (+ 3841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:57979: BlockingRecv error: recv got EOF from 127.1.210.1:57979 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 11:48:27 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:27 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:27.146862 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:27.138529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:33103)
0924 11:48:27.139089 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:27.139105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:27.139216 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:27.139496 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:27.139499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:27.139868 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:27.140045 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:27.140053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.141215 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.141220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:27.142136 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:27.142148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.142964 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.142970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:27.143145 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:27.143713 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:27.143730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:27.146509 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
W20260924 11:48:27.146994 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:27.138751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:33103 (local address 127.1.210.62:45337)
0924 11:48:27.138945 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:27.138978 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:27.139180 (+ 202us) server_negotiation.cc:407] Connection header received
0924 11:48:27.139315 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:27.139318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:27.139377 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:27.139458 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:27.140167 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.141040 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.142285 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.142780 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.142923 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:27.146735 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:33103: BlockingRecv error: recv got EOF from 127.1.210.2:33103 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 11:48:27 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:27 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:27.609402 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:27.595001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:50391)
0924 11:48:27.595574 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:27.595590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:27.595737 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:27.596076 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:27.596080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:27.596496 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:27.596635 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:27.596642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.597691 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.597697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:27.599120 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:27.599134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.599896 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.599902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:27.600074 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:27.605927 (+ 5853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:27.605944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:27.609045 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":257,"threads_started":1}
W20260924 11:48:27.609476 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:27.595230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:50391 (local address 127.1.210.62:45337)
0924 11:48:27.595456 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:27.595481 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:27.595673 (+ 192us) server_negotiation.cc:407] Connection header received
0924 11:48:27.595866 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:27.595873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:27.595935 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:27.596046 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:27.596775 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.597556 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.599242 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.599722 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.599866 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:27.609248 (+ 9382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:50391: BlockingRecv error: recv got EOF from 127.1.210.3:50391 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 11:48:27 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:27 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:27.707242 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:27.699092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:37327)
0924 11:48:27.699624 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:27.699639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:27.699747 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:27.700107 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:27.700110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:27.700460 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:27.700617 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:27.700625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.701627 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.701631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:27.702560 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:27.702571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.703216 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.703220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:27.703332 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:27.703962 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:27.704002 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:27.707009 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":308,"threads_started":1}
W20260924 11:48:27.707365 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:27.699283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:37327 (local address 127.1.210.62:45337)
0924 11:48:27.699442 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:27.699446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:27.699820 (+ 374us) server_negotiation.cc:407] Connection header received
0924 11:48:27.699870 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:27.699874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:27.699948 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:27.700058 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:27.700733 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.701511 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.702673 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:27.703090 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:27.703178 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:27.707168 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:37327: BlockingRecv error: recv got EOF from 127.1.210.1:37327 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:48:28 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:28 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:28.156237 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:28.147940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:56639)
0924 11:48:28.148518 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:28.148534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:28.148644 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:28.148949 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:28.148957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:28.149412 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:28.149580 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:28.149589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.150924 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.150928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:28.151838 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:28.151848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.152502 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.152506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:28.152625 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:28.153196 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:28.153212 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:28.155893 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":311,"threads_started":1}
W20260924 11:48:28.156319 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:28.148001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:56639 (local address 127.1.210.62:45337)
0924 11:48:28.148159 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:28.148164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:28.148607 (+ 443us) server_negotiation.cc:407] Connection header received
0924 11:48:28.148739 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:28.148742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:28.148800 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:28.148879 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:28.149752 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.150805 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.151965 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.152368 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.152461 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:28.156089 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:56639: BlockingRecv error: recv got EOF from 127.1.210.2:56639 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:48:28 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:28 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:28.618359 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:28.610338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:59073)
0924 11:48:28.610917 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:28.610933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:28.611059 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:28.611311 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:28.611315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:28.611698 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:28.611852 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:28.611860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.612753 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.612756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:28.613686 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:28.613696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.614434 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.614440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:28.614633 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:28.615260 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:28.615277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:28.618025 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
W20260924 11:48:28.618391 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:28.610415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:59073 (local address 127.1.210.62:45337)
0924 11:48:28.610581 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:28.610585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:28.611008 (+ 423us) server_negotiation.cc:407] Connection header received
0924 11:48:28.611137 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:28.611140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:28.611194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:28.611267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:28.611970 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.612644 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.613796 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.614236 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.614407 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:28.618193 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:59073: BlockingRecv error: recv got EOF from 127.1.210.3:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:48:28 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:28 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:28.716652 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:28.708363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:36079)
0924 11:48:28.708898 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:28.708916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:28.709049 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:28.709430 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:28.709433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:28.709845 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:28.709999 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:28.710008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.711004 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.711008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:28.711969 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:28.711980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.712619 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.712624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:28.712751 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:28.713377 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:28.713396 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:28.716415 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":307,"threads_started":1}
W20260924 11:48:28.716899 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:28.708479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:36079 (local address 127.1.210.62:45337)
0924 11:48:28.708740 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:28.708746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:28.709002 (+ 256us) server_negotiation.cc:407] Connection header received
0924 11:48:28.709193 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:28.709197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:28.709285 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:28.709362 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:28.710136 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.710851 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.712090 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:28.712476 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:28.712570 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:28.716606 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:36079: BlockingRecv error: recv got EOF from 127.1.210.1:36079 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 11:48:29 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:29 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:29.165122 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:29.157329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:39883)
0924 11:48:29.157908 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:29.157923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:29.158032 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:29.158306 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:29.158309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:29.158704 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:29.158852 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:29.158860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.159786 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.159790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:29.160689 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:29.160699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.161385 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.161389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:29.161511 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:29.162047 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:29.162063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:29.164776 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":360,"threads_started":1}
W20260924 11:48:29.165269 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:29.157408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:39883 (local address 127.1.210.62:45337)
0924 11:48:29.157611 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:29.157620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:29.157997 (+ 377us) server_negotiation.cc:407] Connection header received
0924 11:48:29.158131 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:29.158134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:29.158193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:29.158263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:29.158972 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.159664 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.160800 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.161266 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.161355 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:29.165004 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:39883: BlockingRecv error: recv got EOF from 127.1.210.2:39883 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:48:29 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:29 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:29.627735 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:29.619496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:35017)
0924 11:48:29.619977 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:29.619993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:29.620112 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:29.620503 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:29.620507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:29.620901 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:29.621140 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:29.621154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.622477 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.622481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:29.623472 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:29.623483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.624146 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.624150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:29.624268 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:29.624809 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:29.624825 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:29.627504 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":286,"threads_started":1}
W20260924 11:48:29.627969 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:29.619698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:35017 (local address 127.1.210.62:45337)
0924 11:48:29.619950 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:29.619957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:29.620081 (+ 124us) server_negotiation.cc:407] Connection header received
0924 11:48:29.620225 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:29.620229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:29.620322 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:29.620432 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:29.621309 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.622353 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.623593 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.623996 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.624093 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:29.627710 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:35017: BlockingRecv error: recv got EOF from 127.1.210.3:35017 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 24 11:48:29 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:29 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:29.725196 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:29.717647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:43695)
0924 11:48:29.718140 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:29.718166 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:29.718275 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:29.718576 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:29.718579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:29.718954 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:29.719119 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:29.719127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.720067 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.720070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:29.720930 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:29.720940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.721579 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.721583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:29.721689 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:29.722244 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:29.722259 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:29.724858 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
W20260924 11:48:29.725489 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:29.717737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:43695 (local address 127.1.210.62:45337)
0924 11:48:29.717911 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:29.717915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:29.718242 (+ 327us) server_negotiation.cc:407] Connection header received
0924 11:48:29.718374 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:29.718377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:29.718448 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:29.718543 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:29.719236 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.719964 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.721042 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:29.721446 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:29.721542 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:29.725125 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:43695: BlockingRecv error: recv got EOF from 127.1.210.1:43695 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:48:30 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:30 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:30.174163 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:30.166057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:41707)
0924 11:48:30.166610 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:30.166625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:30.166735 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:30.167087 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:30.167092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:30.167506 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:30.167776 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:30.167786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.168996 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.169000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:30.169898 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:30.169909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.170589 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.170593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:30.170726 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:30.171295 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:30.171311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:30.173936 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":312,"threads_started":1}
W20260924 11:48:30.174347 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:30.166151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:41707 (local address 127.1.210.62:45337)
0924 11:48:30.166315 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:30.166320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:30.166699 (+ 379us) server_negotiation.cc:407] Connection header received
0924 11:48:30.166833 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:30.166838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:30.166915 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:30.167023 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:30.167922 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.168883 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.170025 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.170444 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.170549 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:30.174121 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:41707: BlockingRecv error: recv got EOF from 127.1.210.2:41707 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 11:48:30 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:30 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:30.636875 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:30.628642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:42143)
0924 11:48:30.629229 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:30.629244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:30.629389 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:30.629714 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:30.629718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:30.630144 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:30.630307 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:30.630315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.631285 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.631291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:30.632248 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:30.632266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.632947 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.632953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:30.633163 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:30.633699 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:30.633715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:30.636506 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
W20260924 11:48:30.636921 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:30.628758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:42143 (local address 127.1.210.62:45337)
0924 11:48:30.628957 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:30.628965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:30.629325 (+ 360us) server_negotiation.cc:407] Connection header received
0924 11:48:30.629514 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:30.629524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:30.629588 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:30.629689 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:30.630447 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.631155 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.632387 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.632797 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.632932 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:30.636731 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:42143: BlockingRecv error: recv got EOF from 127.1.210.3:42143 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 11:48:30 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:30 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:30.734602 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:30.726088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:51891)
0924 11:48:30.726590 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:30.726605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:30.726760 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:30.727115 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:30.727119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:30.727535 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:30.727680 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:30.727688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.728704 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.728712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:30.729774 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:30.729785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.730817 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.730822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:30.730993 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:30.731618 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:30.731635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:30.734365 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":294,"threads_started":1}
W20260924 11:48:30.734810 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:30.726146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:51891 (local address 127.1.210.62:45337)
0924 11:48:30.726348 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:30.726354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:30.726693 (+ 339us) server_negotiation.cc:407] Connection header received
0924 11:48:30.726890 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:30.726895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:30.726977 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:30.727085 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:30.727819 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.728570 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.729908 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:30.730693 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:30.730809 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:30.734540 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:51891: BlockingRecv error: recv got EOF from 127.1.210.1:51891 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:48:31 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:31 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:31.183487 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:31.175082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:46789)
0924 11:48:31.175621 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:31.175636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:31.175744 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:31.176083 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:31.176088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:31.176645 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:31.176839 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:31.176853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.177993 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.177997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:31.178907 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:31.178918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.179555 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.179561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:31.179694 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:31.180302 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:31.180318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:31.183144 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":301,"threads_started":1}
W20260924 11:48:31.183547 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:31.175162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:46789 (local address 127.1.210.62:45337)
0924 11:48:31.175319 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:31.175323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:31.175707 (+ 384us) server_negotiation.cc:407] Connection header received
0924 11:48:31.175836 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:31.175839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:31.175896 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:31.175976 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:31.176992 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.177863 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.179024 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.179393 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.179486 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:31.183316 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:46789: BlockingRecv error: recv got EOF from 127.1.210.2:46789 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 11:48:31 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:31 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:31.645390 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:31.637723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:42213)
0924 11:48:31.638263 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:31.638278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:31.638384 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:31.638661 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:31.638665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:31.639020 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:31.639166 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:31.639174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.640199 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.640203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:31.641106 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:31.641116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.641779 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.641783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:31.641903 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:31.642438 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:31.642454 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:31.645021 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":338,"threads_started":1}
W20260924 11:48:31.645484 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:31.637760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:42213 (local address 127.1.210.62:45337)
0924 11:48:31.637905 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:31.637910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:31.638346 (+ 436us) server_negotiation.cc:407] Connection header received
0924 11:48:31.638463 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:31.638467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:31.638524 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:31.638603 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:31.639361 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.640082 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.641234 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.641637 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.641735 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:31.645224 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:42213: BlockingRecv error: recv got EOF from 127.1.210.3:42213 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 11:48:31 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:31 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:31.743798 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:31.735391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:40083)
0924 11:48:31.735909 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:31.735925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:31.736037 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:31.736333 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:31.736336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:31.736700 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:31.736851 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:31.736859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.737816 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.737819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:31.738991 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:31.739023 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.739935 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.739960 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:31.740119 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:31.740799 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:31.740816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:31.743458 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
W20260924 11:48:31.743862 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:31.735475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:40083 (local address 127.1.210.62:45337)
0924 11:48:31.735644 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:31.735649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:31.735998 (+ 349us) server_negotiation.cc:407] Connection header received
0924 11:48:31.736133 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:31.736137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:31.736209 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:31.736290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:31.736968 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.737697 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.739151 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:31.739734 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:31.739913 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:31.743645 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:40083: BlockingRecv error: recv got EOF from 127.1.210.1:40083 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:48:32 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:32 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:32.192759 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:32.184463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:38525)
0924 11:48:32.185038 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:32.185095 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:32.185230 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:32.185564 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:32.185569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:32.186111 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:32.186336 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:32.186348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.187438 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.187443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:32.188357 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:32.188368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.189146 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.189154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:32.189323 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:32.189862 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:32.189878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:32.192505 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":283,"threads_started":1}
W20260924 11:48:32.192926 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:32.184576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:38525 (local address 127.1.210.62:45337)
0924 11:48:32.184773 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:32.184781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:32.185162 (+ 381us) server_negotiation.cc:407] Connection header received
0924 11:48:32.185362 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:32.185368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:32.185436 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:32.185538 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:32.186455 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.187304 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.188505 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.188937 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.189064 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:32.192711 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:38525: BlockingRecv error: recv got EOF from 127.1.210.2:38525 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:48:32 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:32 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:32.659463 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:32.646352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:38991)
0924 11:48:32.646853 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:32.646868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:32.646977 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:32.647258 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:32.647261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:32.647632 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:32.647780 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:32.647788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.648731 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.648734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:32.649708 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:32.649720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.650358 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.650362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:32.650505 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:32.656423 (+ 5918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:32.656441 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:32.659237 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
W20260924 11:48:32.659669 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:32.646373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:38991 (local address 127.1.210.62:45337)
0924 11:48:32.646535 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:32.646540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:32.646940 (+ 400us) server_negotiation.cc:407] Connection header received
0924 11:48:32.647078 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:32.647082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:32.647138 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:32.647215 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:32.647896 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.648609 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.649829 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.650214 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.650301 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:32.659470 (+ 9169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:38991: BlockingRecv error: recv got EOF from 127.1.210.3:38991 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:48:32 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:32 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:32.752652 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:32.744716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:52667)
0924 11:48:32.745193 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:32.745208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:32.745321 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:32.745603 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:32.745606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:32.745939 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:32.746107 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:32.746116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.747083 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.747086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:32.747973 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:32.747983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.748931 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.748937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:32.749111 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:32.749803 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:32.749820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:32.752412 (+ 2592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
W20260924 11:48:32.752883 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:32.744855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:52667 (local address 127.1.210.62:45337)
0924 11:48:32.745159 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:32.745166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:32.745283 (+ 117us) server_negotiation.cc:407] Connection header received
0924 11:48:32.745408 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:32.745413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:32.745476 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:32.745553 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:32.746230 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.746959 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.748118 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:32.748777 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:32.748902 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:32.752627 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:52667: BlockingRecv error: recv got EOF from 127.1.210.1:52667 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 24 11:48:33 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:33 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:33.201522 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:33.193760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:54375)
0924 11:48:33.194392 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:33.194414 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:33.194552 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:33.194820 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:33.194823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:33.195175 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:33.195341 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:33.195350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.196282 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.196285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:33.197174 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:33.197185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.197772 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.197776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:33.197884 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:33.198440 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:33.198475 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:33.201173 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":384,"threads_started":1}
W20260924 11:48:33.201542 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:33.193832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:54375 (local address 127.1.210.62:45337)
0924 11:48:33.194022 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:33.194029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:33.194508 (+ 479us) server_negotiation.cc:407] Connection header received
0924 11:48:33.194644 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:33.194647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:33.194703 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:33.194780 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:33.195471 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.196163 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.197291 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.197659 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.197748 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:33.201343 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:54375: BlockingRecv error: recv got EOF from 127.1.210.2:54375 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 11:48:33 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:33 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.3@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:33.668675 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:33.660416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.3:59321)
0924 11:48:33.660894 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:33.660910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:33.661020 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:33.661329 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:33.661332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:33.661764 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:33.661979 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:33.661991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.663387 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.663390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:33.664297 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:33.664308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.664938 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.664942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:33.665051 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:33.665623 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:33.665638 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:33.668281 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
W20260924 11:48:33.668704 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:33.660543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:59321 (local address 127.1.210.62:45337)
0924 11:48:33.660739 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:33.660743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:33.660983 (+ 240us) server_negotiation.cc:407] Connection header received
0924 11:48:33.661151 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:33.661155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:33.661210 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:33.661290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:33.662154 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.663246 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.664409 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.664829 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.664917 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:33.668471 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.3:59321: BlockingRecv error: recv got EOF from 127.1.210.3:59321 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 11:48:33 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:33 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.1@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:33.761896 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:33.753882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.1:48885)
0924 11:48:33.754350 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:33.754364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:33.754507 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:33.754877 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:33.754881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:33.755396 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:33.755632 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:33.755642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.756631 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.756639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:33.757590 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:33.757601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.758322 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.758327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:33.758504 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:33.759014 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:33.759030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:33.761646 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
W20260924 11:48:33.762030 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:33.753973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:48885 (local address 127.1.210.62:45337)
0924 11:48:33.754173 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:33.754182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:33.754443 (+ 261us) server_negotiation.cc:407] Connection header received
0924 11:48:33.754634 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:33.754639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:33.754707 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:33.754825 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:33.755768 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.756499 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.757728 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:33.758180 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:33.758312 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:33.761833 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.1:48885: BlockingRecv error: recv got EOF from 127.1.210.1:48885 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 11:48:34 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:48:34 dist-test-slave-dd32 krb5kdc[10079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.2@KRBTEST.COM for oryx/127.1.210.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:48:34.210140 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:34.202584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:45337 (local address 127.1.210.2:52945)
0924 11:48:34.203042 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:34.203058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:34.203166 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:34.203450 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:34.203453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:34.203816 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:34.203991 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:34.203998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.205010 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.205013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:34.205947 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:34.205958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.206553 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.206556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:34.206671 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:34.207292 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:34.207308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:34.209902 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.62:45337: Server oryx/127.1.210.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":255,"threads_started":1}
W20260924 11:48:34.210315 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:34.202632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:52945 (local address 127.1.210.62:45337)
0924 11:48:34.202845 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:34.202854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:34.203134 (+ 280us) server_negotiation.cc:407] Connection header received
0924 11:48:34.203257 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:34.203261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:34.203330 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:34.203404 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:34.204102 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.204902 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.206061 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.206442 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.206528 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:34.210072 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.2:52945: BlockingRecv error: recv got EOF from 127.1.210.2:52945 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:48:34.492619 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10666
I20260924 11:48:34.503798 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:34.493655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.1.210.1:43723)
0924 11:48:34.494148 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:34.494156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:34.494181 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:48:34.494236 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:34.494242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:34.494461 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:34.494627 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:34.495908 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.496915 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.498119 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.498660 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.498757 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:34.501046 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:34.501054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:34.501110 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:34.501151 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:34.503343 (+ 2192us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:48:34.503570 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32968: Request ticket server kudu/127.1.210.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
W20260924 11:48:34.503913 11127 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32968: Request ticket server kudu/127.1.210.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:48:34.503957 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10666: Not authorized: Client connection negotiation failed: client connection to 127.1.210.1:43723: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:48:34.504169 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10666
I20260924 11:48:34.520251 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10818
I20260924 11:48:34.532006 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:34.521260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.1.210.2:42633)
0924 11:48:34.521873 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:34.521880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:34.521911 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:48:34.521971 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:34.521978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:34.522220 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:34.522377 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:34.523529 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.524539 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.525792 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.526317 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.526415 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:34.528676 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:34.528683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:34.528697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:34.528726 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:34.531522 (+ 2796us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:48:34.531763 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50512: Request ticket server kudu/127.1.210.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":302,"threads_started":1}
W20260924 11:48:34.532124 11128 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50512: Request ticket server kudu/127.1.210.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:48:34.532152 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10818: Not authorized: Client connection negotiation failed: client connection to 127.1.210.2:42633: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:48:34.532298 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10818
I20260924 11:48:34.548343 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10949
I20260924 11:48:34.559386 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:34.549239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.1.210.3:39565)
0924 11:48:34.549831 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:34.549838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:34.549863 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:48:34.549921 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:34.549927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:34.550165 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:34.550311 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:34.551513 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.552530 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.553800 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:34.554345 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:34.554434 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:34.556663 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:34.556671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:34.556683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:34.556723 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:34.558916 (+ 2193us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:48:34.559151 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37290: Request ticket server kudu/127.1.210.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":338,"threads_started":1}
W20260924 11:48:34.559527 11129 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37290: Request ticket server kudu/127.1.210.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:48:34.559523 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10949: Not authorized: Client connection negotiation failed: client connection to 127.1.210.3:39565: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:48:34.559803 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10949
I20260924 11:48:34.576400 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 10597
I20260924 11:48:34.577704 10659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:34.690104 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 10597
2026-09-24T11:48:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20880 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T11:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:34Z Disabled control of system clock
I20260924 11:48:34.763432 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37327
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 11:48:35.079082 11134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:35.079360 11134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:35.079495 11134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:35.086673 11134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:35.086771 11134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:35.086853 11134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:35.086953 11134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:35.087039 11134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:35.096020 11134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:37327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:35.097302 11134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:35.099044 11134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:35.110915 11139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:35.110989 11140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:35.111016 11142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:35.111593 11134 server_base.cc:1034] running on GCE node
I20260924 11:48:35.112293 11134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:35.113917 11134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:35.115154 11134 hybrid_clock.cc:648] HybridClock initialized: now 1790250515115113 us; error 45 us; skew 500 ppm
I20260924 11:48:35.115506 11134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:35.117650 11134 webserver.cc:460] Webserver started at http://127.1.210.62:35843/ using document root <none> and password file <none>
I20260924 11:48:35.118297 11134 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:35.118423 11134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:35.118728 11134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:35.121539 11134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "504c3a4ad8a2495a9884e87eaf0565fc"
format_stamp: "Formatted at 2026-09-24 11:48:35 on dist-test-slave-dd32"
server_key: "37333e51cfa29bde97645cfe892c7990"
server_key_iv: "f2a3c264d9f93a652866cafd368f4287"
server_key_version: "encryptionkey@0"
I20260924 11:48:35.122284 11134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "504c3a4ad8a2495a9884e87eaf0565fc"
format_stamp: "Formatted at 2026-09-24 11:48:35 on dist-test-slave-dd32"
server_key: "37333e51cfa29bde97645cfe892c7990"
server_key_iv: "f2a3c264d9f93a652866cafd368f4287"
server_key_version: "encryptionkey@0"
I20260924 11:48:35.126991 11134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:35.130100 11148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:35.131300 11134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:48:35.131479 11134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "504c3a4ad8a2495a9884e87eaf0565fc"
format_stamp: "Formatted at 2026-09-24 11:48:35 on dist-test-slave-dd32"
server_key: "37333e51cfa29bde97645cfe892c7990"
server_key_iv: "f2a3c264d9f93a652866cafd368f4287"
server_key_version: "encryptionkey@0"
I20260924 11:48:35.131675 11134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:35.154496 11134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:35.155290 11134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:35.155577 11134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:35.173174 11134 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:37327
I20260924 11:48:35.173204 11199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:37327 every 8 connection(s)
I20260924 11:48:35.174510 11134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:48:35.178089 11200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:35.181597 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11134
I20260924 11:48:35.181779 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:48:35.182583 1864 external_mini_cluster.cc:1428] Setting key 1d19147be588b1f4bd4e76d4a30653ba
I20260924 11:48:35.190521 11200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc: Bootstrap starting.
I20260924 11:48:35.193629 11200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:35.194697 11200 log.cc:824] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:35.195689 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:35.184655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.1.210.62:37327)
0924 11:48:35.185663 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:35.185674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:35.185713 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:48:35.187261 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:35.187281 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:35.187715 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:35.188116 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:35.188690 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:35.189959 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:35.191363 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:35.192018 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:35.192237 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:35.193400 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:35.193407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:35.193420 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:35.193986 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:35.194079 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:35.194106 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:35.194234 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:35.194454 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 11:48:35.194732 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":435,"threads_started":1}
I20260924 11:48:35.197530 11200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc: No bootstrap required, opened a new log
I20260924 11:48:35.205193 11200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37327 } }
I20260924 11:48:35.205509 11200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:35.205641 11200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c3a4ad8a2495a9884e87eaf0565fc, State: Initialized, Role: FOLLOWER
I20260924 11:48:35.206240 11200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37327 } }
I20260924 11:48:35.206457 11200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:48:35.206616 11200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:48:35.206780 11200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:35.208757 11200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37327 } }
I20260924 11:48:35.209273 11200 leader_election.cc:302] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 504c3a4ad8a2495a9884e87eaf0565fc; no voters:
I20260924 11:48:35.209817 11200 leader_election.cc:288] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:48:35.210000 11205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:35.210721 11205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [term 1 LEADER]: Becoming Leader. State: Replica: 504c3a4ad8a2495a9884e87eaf0565fc, State: Running, Role: LEADER
I20260924 11:48:35.211258 11205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37327 } }
I20260924 11:48:35.211771 11200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:35.214270 11207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 504c3a4ad8a2495a9884e87eaf0565fc. Latest consensus state: current_term: 1 leader_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37327 } } }
I20260924 11:48:35.214551 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504c3a4ad8a2495a9884e87eaf0565fc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37327 } } }
I20260924 11:48:35.214781 11207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:35.214871 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:35.215972 11214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:35.221285 11214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:35.229315 11214 catalog_manager.cc:1347] Generated new cluster ID: ee11846119d14a90a8995c1ee4bee952
I20260924 11:48:35.229485 11214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:35.236685 11214 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:48:35.238389 11214 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:35.248152 11214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 504c3a4ad8a2495a9884e87eaf0565fc: Generated new TSK 0
I20260924 11:48:35.249153 11214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:48:35.255736 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37327
--builtin_ntp_servers=127.1.210.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 11:48:35.575048 11224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:35.575320 11224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:35.575402 11224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:35.582386 11224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:35.582476 11224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:35.582680 11224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:35.591305 11224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:35.592549 11224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:35.594275 11224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:35.606755 11229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:35.606690 11230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:35.606721 11232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:35.607306 11224 server_base.cc:1034] running on GCE node
I20260924 11:48:35.607920 11224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:35.608726 11224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:35.609946 11224 hybrid_clock.cc:648] HybridClock initialized: now 1790250515609899 us; error 61 us; skew 500 ppm
I20260924 11:48:35.610260 11224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:35.612442 11224 webserver.cc:460] Webserver started at http://127.1.210.1:33231/ using document root <none> and password file <none>
I20260924 11:48:35.613153 11224 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:35.613291 11224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:35.613617 11224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:35.616304 11224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "edfed7b916404fb9a0ff97b762980f3f"
format_stamp: "Formatted at 2026-09-24 11:48:35 on dist-test-slave-dd32"
server_key: "ed95c4e62666f386ad91ed268099afa4"
server_key_iv: "d470db5f856f150219301f8594517b47"
server_key_version: "encryptionkey@0"
I20260924 11:48:35.617044 11224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "edfed7b916404fb9a0ff97b762980f3f"
format_stamp: "Formatted at 2026-09-24 11:48:35 on dist-test-slave-dd32"
server_key: "ed95c4e62666f386ad91ed268099afa4"
server_key_iv: "d470db5f856f150219301f8594517b47"
server_key_version: "encryptionkey@0"
I20260924 11:48:35.621814 11224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:48:35.625131 11238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:35.626346 11224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:48:35.626510 11224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "edfed7b916404fb9a0ff97b762980f3f"
format_stamp: "Formatted at 2026-09-24 11:48:35 on dist-test-slave-dd32"
server_key: "ed95c4e62666f386ad91ed268099afa4"
server_key_iv: "d470db5f856f150219301f8594517b47"
server_key_version: "encryptionkey@0"
I20260924 11:48:35.626727 11224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:35.645291 11224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:35.646076 11224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:35.646342 11224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:35.647243 11224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:35.648591 11224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:35.648730 11224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:35.648847 11224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:35.648933 11224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:35.681277 11224 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:39569
I20260924 11:48:35.681327 11350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:39569 every 8 connection(s)
I20260924 11:48:35.682626 11224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:35.689005 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11224
I20260924 11:48:35.689256 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:48:35.690068 1864 external_mini_cluster.cc:1428] Setting key c7bfeecc0c4cd9ac87bbc70caab3858e
I20260924 11:48:35.693099 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37327
--builtin_ntp_servers=127.1.210.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 11:48:35.696744 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:35.685208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:60979 (local address 127.1.210.62:37327)
0924 11:48:35.685410 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:35.685415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:35.687167 (+ 1752us) server_negotiation.cc:407] Connection header received
0924 11:48:35.688979 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:35.688983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:35.689047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:35.689196 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:35.690582 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:35.691423 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:35.693708 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:35.694093 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:35.694191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:35.695798 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:35.695806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:35.695813 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:35.695836 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:35.695873 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:35.695890 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:35.695971 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:35.696263 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 11:48:35.696368 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:48:35.698812 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:35.685759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37327 (local address 127.1.210.1:60979)
0924 11:48:35.687044 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:35.687080 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:35.688690 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:35.689466 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:35.689479 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:35.689604 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:35.690421 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:35.690446 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:35.691579 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:35.691583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:35.693547 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:35.693569 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:35.694320 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:35.694329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:35.694668 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:35.695444 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:48:35.695449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:35.695501 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:48:35.695595 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:35.695993 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:35.696005 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:35.696016 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:48:35.696998 (+ 982us) client_negotiation.cc:240] Negotiation successful
0924 11:48:35.697438 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":371,"threads_started":1}
I20260924 11:48:35.700390 11351 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37327
I20260924 11:48:35.700693 11351 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:35.701478 11351 heartbeater.cc:511] Master 127.1.210.62:37327 requested a full tablet report, sending...
I20260924 11:48:35.703547 11165 ts_manager.cc:194] Registered new tserver with Master: edfed7b916404fb9a0ff97b762980f3f (127.1.210.1:39569)
I20260924 11:48:35.705721 11165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.1:60979
W20260924 11:48:36.009272 11354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:36.009553 11354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:36.009637 11354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:36.016709 11354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:36.016798 11354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:36.016894 11354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:36.025470 11354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:36.026576 11354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:36.028136 11354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:36.040953 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:36.040989 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:36.041487 11362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:36.041604 11354 server_base.cc:1034] running on GCE node
I20260924 11:48:36.042258 11354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:36.043102 11354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:36.044312 11354 hybrid_clock.cc:648] HybridClock initialized: now 1790250516044269 us; error 57 us; skew 500 ppm
I20260924 11:48:36.044612 11354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:36.046754 11354 webserver.cc:460] Webserver started at http://127.1.210.2:45119/ using document root <none> and password file <none>
I20260924 11:48:36.047390 11354 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:36.047523 11354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:36.047823 11354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:36.050446 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "fcc2a210ebe445b1a8f8f87adc78f66b"
format_stamp: "Formatted at 2026-09-24 11:48:36 on dist-test-slave-dd32"
server_key: "0d4a1fd42de832f04383cbef3aedfedd"
server_key_iv: "50ecc79c30c0c3c26d3b27de32bb2361"
server_key_version: "encryptionkey@0"
I20260924 11:48:36.051188 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "fcc2a210ebe445b1a8f8f87adc78f66b"
format_stamp: "Formatted at 2026-09-24 11:48:36 on dist-test-slave-dd32"
server_key: "0d4a1fd42de832f04383cbef3aedfedd"
server_key_iv: "50ecc79c30c0c3c26d3b27de32bb2361"
server_key_version: "encryptionkey@0"
I20260924 11:48:36.056325 11354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 11:48:36.059389 11368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:36.060555 11354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:48:36.060767 11354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "fcc2a210ebe445b1a8f8f87adc78f66b"
format_stamp: "Formatted at 2026-09-24 11:48:36 on dist-test-slave-dd32"
server_key: "0d4a1fd42de832f04383cbef3aedfedd"
server_key_iv: "50ecc79c30c0c3c26d3b27de32bb2361"
server_key_version: "encryptionkey@0"
I20260924 11:48:36.061017 11354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:36.072391 11354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:36.073210 11354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:36.073469 11354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:36.074373 11354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:36.075655 11354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:36.075767 11354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:36.075894 11354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:36.075987 11354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:36.107679 11354 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:45023
I20260924 11:48:36.107764 11480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:45023 every 8 connection(s)
I20260924 11:48:36.109174 11354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:36.117488 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11354
I20260924 11:48:36.117688 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:48:36.118124 1864 external_mini_cluster.cc:1428] Setting key 276035fe07c218da69a9e1c510c7d4f7
I20260924 11:48:36.120793 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37327
--builtin_ntp_servers=127.1.210.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 11:48:36.122162 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.111828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:52195 (local address 127.1.210.62:37327)
0924 11:48:36.111987 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:36.111991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:36.113411 (+ 1420us) server_negotiation.cc:407] Connection header received
0924 11:48:36.115200 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:36.115206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:36.115289 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:36.115424 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:36.116553 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.117439 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.119290 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.119859 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.119990 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.121386 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:36.121393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:36.121402 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:36.121418 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:36.121447 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:36.121458 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:36.121547 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:36.121754 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 11:48:36.121841 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:48:36.123113 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.112277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37327 (local address 127.1.210.2:52195)
0924 11:48:36.113264 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:36.113299 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:36.114918 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:36.115711 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:36.115718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:36.115824 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:36.116393 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:36.116409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.117647 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.117653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:36.119167 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:36.119179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.120074 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.120080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:36.120309 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.121031 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:48:36.121041 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:36.121142 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:48:36.121210 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:36.121545 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:36.121557 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:36.121567 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:48:36.121753 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 11:48:36.122095 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":357,"threads_started":1}
I20260924 11:48:36.124493 11481 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37327
I20260924 11:48:36.124768 11481 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:36.125531 11481 heartbeater.cc:511] Master 127.1.210.62:37327 requested a full tablet report, sending...
I20260924 11:48:36.126825 11165 ts_manager.cc:194] Registered new tserver with Master: fcc2a210ebe445b1a8f8f87adc78f66b (127.1.210.2:45023)
I20260924 11:48:36.127610 11165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.2:52195
W20260924 11:48:36.449638 11484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:36.449992 11484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:36.450152 11484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:36.458227 11484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:36.458370 11484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:36.458542 11484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:36.467875 11484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:36.469213 11484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:36.471057 11484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:36.483870 11489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:36.483840 11492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:36.483947 11490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:36.484805 11484 server_base.cc:1034] running on GCE node
I20260924 11:48:36.485584 11484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:36.486528 11484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:36.487761 11484 hybrid_clock.cc:648] HybridClock initialized: now 1790250516487728 us; error 43 us; skew 500 ppm
I20260924 11:48:36.488116 11484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:36.490337 11484 webserver.cc:460] Webserver started at http://127.1.210.3:35991/ using document root <none> and password file <none>
I20260924 11:48:36.491008 11484 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:36.491139 11484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:36.491472 11484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:36.494234 11484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "02b38d6f1d1843a98132b30b1a497de8"
format_stamp: "Formatted at 2026-09-24 11:48:36 on dist-test-slave-dd32"
server_key: "2685e79c477483f05230c39ea7c7d94b"
server_key_iv: "6668536831605b942ecf1952f45e57a1"
server_key_version: "encryptionkey@0"
I20260924 11:48:36.495025 11484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "02b38d6f1d1843a98132b30b1a497de8"
format_stamp: "Formatted at 2026-09-24 11:48:36 on dist-test-slave-dd32"
server_key: "2685e79c477483f05230c39ea7c7d94b"
server_key_iv: "6668536831605b942ecf1952f45e57a1"
server_key_version: "encryptionkey@0"
I20260924 11:48:36.499886 11484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 11:48:36.503134 11498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:36.504268 11484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 11:48:36.504446 11484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "02b38d6f1d1843a98132b30b1a497de8"
format_stamp: "Formatted at 2026-09-24 11:48:36 on dist-test-slave-dd32"
server_key: "2685e79c477483f05230c39ea7c7d94b"
server_key_iv: "6668536831605b942ecf1952f45e57a1"
server_key_version: "encryptionkey@0"
I20260924 11:48:36.504652 11484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:36.520555 11484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:36.521416 11484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:36.521744 11484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:36.522687 11484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:36.524000 11484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:36.524123 11484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:36.524250 11484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:36.524335 11484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:36.557180 11484 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:34353
I20260924 11:48:36.557271 11610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:34353 every 8 connection(s)
I20260924 11:48:36.558542 11484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:36.562649 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11484
I20260924 11:48:36.562839 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:48:36.563519 1864 external_mini_cluster.cc:1428] Setting key 0cafcdb66d5ea9da781ae9b48dedf361
I20260924 11:48:36.572844 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.561098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:38899 (local address 127.1.210.62:37327)
0924 11:48:36.561307 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:36.561313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:36.565091 (+ 3778us) server_negotiation.cc:407] Connection header received
0924 11:48:36.567221 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:36.567225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:36.567287 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:36.567367 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:36.568372 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.569048 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.570553 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.570931 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.571020 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.572257 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:36.572261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:36.572264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:36.572279 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:36.572304 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:36.572315 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:36.572378 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:36.572566 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 11:48:36.572629 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:48:36.573807 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.561692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37327 (local address 127.1.210.3:38899)
0924 11:48:36.564926 (+ 3234us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:36.564971 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:36.566977 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:36.567578 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:36.567586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:36.567680 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:36.568236 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:36.568252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.569235 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.569238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:36.570434 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:36.570445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.571102 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.571107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:36.571294 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.571984 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:48:36.571987 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:36.572021 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:48:36.572097 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:36.572395 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:36.572402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:36.572407 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:48:36.572540 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 11:48:36.572812 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"thread_start_us":2463,"threads_started":1}
I20260924 11:48:36.575099 11611 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37327
I20260924 11:48:36.575338 11611 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:36.576030 11611 heartbeater.cc:511] Master 127.1.210.62:37327 requested a full tablet report, sending...
I20260924 11:48:36.577317 11165 ts_manager.cc:194] Registered new tserver with Master: 02b38d6f1d1843a98132b30b1a497de8 (127.1.210.3:34353)
I20260924 11:48:36.578100 11165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.3:38899
I20260924 11:48:36.578980 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:48:36.584779 11203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:36.583398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.1.210.62:37327)
0924 11:48:36.583540 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:36.583544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:36.583983 (+ 439us) server_negotiation.cc:407] Connection header received
0924 11:48:36.584108 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:36.584112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:36.584202 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:36.584281 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:36.584599 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55618: BlockingRecv error: recv got EOF from 127.0.0.1:55618 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:48:36.585520 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11224
I20260924 11:48:36.592346 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.586424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.1.210.1:39569)
0924 11:48:36.586869 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:36.586876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:36.586910 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:48:36.586967 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:36.586972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:36.587148 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:36.587263 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:36.587577 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.588505 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.589701 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.590211 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.590302 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.591177 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:36.591184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:36.591195 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:36.591706 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:36.591746 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:36.591768 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:36.591850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:36.591956 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 11:48:36.592063 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":227,"threads_started":1}
I20260924 11:48:36.593518 11346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:36.675021 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11224
I20260924 11:48:36.692014 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11354
I20260924 11:48:36.699738 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.692865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48716 (local address 127.1.210.2:45023)
0924 11:48:36.693409 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:36.693416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:36.693440 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:48:36.693503 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:36.693509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:36.693732 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:36.693909 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:36.694301 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.695290 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.696514 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.697162 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.697308 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.698285 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:36.698297 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:36.698318 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:36.698889 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:36.698933 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:36.698952 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:36.699086 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:36.699236 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 11:48:36.699380 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":305,"threads_started":1}
I20260924 11:48:36.700977 11476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:36.781888 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11354
I20260924 11:48:36.797438 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11484
I20260924 11:48:36.805290 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:36.798329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.1.210.3:34353)
0924 11:48:36.798975 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:36.798981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:36.799011 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:48:36.799075 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:36.799082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:36.799301 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:36.799443 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:36.799925 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.800933 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.802186 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:36.802748 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:36.802845 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:36.803807 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:36.803815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:36.803829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:36.804428 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:36.804480 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:36.804499 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:36.804586 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:36.804707 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 11:48:36.804857 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":364,"threads_started":1}
I20260924 11:48:36.806499 11606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:36.888422 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11484
I20260924 11:48:36.903617 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11134
I20260924 11:48:36.904791 11195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:37.007755 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11134
2026-09-24T11:48:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2312 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T11:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:37Z Disabled control of system clock
I20260924 11:48:37.075965 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37425
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:37425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 11:48:37.389437 11630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:37.389700 11630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:37.389780 11630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:37.396695 11630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:37.396795 11630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:37.396842 11630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:37.396893 11630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:37.396936 11630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:37.405670 11630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:37425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:37.406816 11630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:37.408483 11630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:37.419832 11635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:37.419865 11636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:37.419796 11638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:37.422447 11630 server_base.cc:1034] running on GCE node
I20260924 11:48:37.423153 11630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:37.424408 11630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:37.425593 11630 hybrid_clock.cc:648] HybridClock initialized: now 1790250517425555 us; error 55 us; skew 500 ppm
I20260924 11:48:37.425922 11630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:37.428045 11630 webserver.cc:460] Webserver started at http://127.1.210.62:44423/ using document root <none> and password file <none>
I20260924 11:48:37.428697 11630 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:37.428809 11630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:37.429200 11630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:37.431903 11630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "5a99388ae00843ec955d96c0dbd8535a"
format_stamp: "Formatted at 2026-09-24 11:48:37 on dist-test-slave-dd32"
server_key: "a7c6d87727e932dc0324426edf418aef"
server_key_iv: "ae37d17d00ffc425f00dae9714aba266"
server_key_version: "encryptionkey@0"
I20260924 11:48:37.432672 11630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "5a99388ae00843ec955d96c0dbd8535a"
format_stamp: "Formatted at 2026-09-24 11:48:37 on dist-test-slave-dd32"
server_key: "a7c6d87727e932dc0324426edf418aef"
server_key_iv: "ae37d17d00ffc425f00dae9714aba266"
server_key_version: "encryptionkey@0"
I20260924 11:48:37.437533 11630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260924 11:48:37.440707 11644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:37.441991 11630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:37.442181 11630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "5a99388ae00843ec955d96c0dbd8535a"
format_stamp: "Formatted at 2026-09-24 11:48:37 on dist-test-slave-dd32"
server_key: "a7c6d87727e932dc0324426edf418aef"
server_key_iv: "ae37d17d00ffc425f00dae9714aba266"
server_key_version: "encryptionkey@0"
I20260924 11:48:37.442392 11630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:37.447840 11630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:37.448575 11630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:37.448859 11630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:37.466560 11630 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:37425
I20260924 11:48:37.466590 11695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:37425 every 8 connection(s)
I20260924 11:48:37.467865 11630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:48:37.471532 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:37.475031 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11630
I20260924 11:48:37.475270 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:48:37.476073 1864 external_mini_cluster.cc:1428] Setting key 8decf25d0dc318f6290e6844f56ba0c5
I20260924 11:48:37.483820 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:37.478066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.1.210.62:37425)
0924 11:48:37.478995 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:37.479005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:37.479044 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:48:37.480524 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:37.480543 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:37.480958 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:37.481327 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:37.482356 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:37.482361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:37.482376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:37.482389 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:37.482448 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:37.482466 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:37.482644 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:37.482736 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 11:48:37.482995 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":353,"threads_started":1}
I20260924 11:48:37.484442 11696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a: Bootstrap starting.
I20260924 11:48:37.487334 11696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:37.488310 11696 log.cc:824] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:37.490839 11696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a: No bootstrap required, opened a new log
I20260924 11:48:37.497682 11696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a99388ae00843ec955d96c0dbd8535a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37425 } }
I20260924 11:48:37.498028 11696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:37.498131 11696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a99388ae00843ec955d96c0dbd8535a, State: Initialized, Role: FOLLOWER
I20260924 11:48:37.498658 11696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a99388ae00843ec955d96c0dbd8535a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37425 } }
I20260924 11:48:37.498899 11696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:48:37.499034 11696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:48:37.499207 11696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:37.501273 11696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a99388ae00843ec955d96c0dbd8535a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37425 } }
I20260924 11:48:37.501763 11696 leader_election.cc:302] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a99388ae00843ec955d96c0dbd8535a; no voters:
I20260924 11:48:37.502300 11696 leader_election.cc:288] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:48:37.502421 11701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:37.503140 11701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [term 1 LEADER]: Becoming Leader. State: Replica: 5a99388ae00843ec955d96c0dbd8535a, State: Running, Role: LEADER
I20260924 11:48:37.503683 11701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a99388ae00843ec955d96c0dbd8535a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37425 } }
I20260924 11:48:37.504372 11696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:37.506492 11703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a99388ae00843ec955d96c0dbd8535a. Latest consensus state: current_term: 1 leader_uuid: "5a99388ae00843ec955d96c0dbd8535a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a99388ae00843ec955d96c0dbd8535a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37425 } } }
I20260924 11:48:37.507012 11703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:37.506520 11702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a99388ae00843ec955d96c0dbd8535a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a99388ae00843ec955d96c0dbd8535a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37425 } } }
I20260924 11:48:37.507500 11702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:37.508512 11708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:37.512686 11708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:37.521026 11708 catalog_manager.cc:1347] Generated new cluster ID: fea0972b6380463d80498187cb8d439e
I20260924 11:48:37.521210 11708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:37.539445 11708 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:48:37.539914 11708 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:37.548858 11708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a99388ae00843ec955d96c0dbd8535a: Generated new TSK 0
I20260924 11:48:37.549850 11708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:48:37.563761 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37425
--builtin_ntp_servers=127.1.210.20:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 11:48:37.882872 11720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:37.883164 11720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:37.883328 11720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:37.890300 11720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:37.890429 11720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:37.890611 11720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:37.899633 11720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:37.900882 11720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:37.902724 11720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:37.915421 11725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:37.915448 11726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:37.915719 11728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:37.916180 11720 server_base.cc:1034] running on GCE node
I20260924 11:48:37.916788 11720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:37.917701 11720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:37.918949 11720 hybrid_clock.cc:648] HybridClock initialized: now 1790250517918888 us; error 58 us; skew 500 ppm
I20260924 11:48:37.919260 11720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:37.925033 11720 webserver.cc:460] Webserver started at http://127.1.210.1:38117/ using document root <none> and password file <none>
I20260924 11:48:37.925876 11720 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:37.926020 11720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:37.926404 11720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:37.932690 11720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "ef9889d5f89c4d128fe47a70bd39129f"
format_stamp: "Formatted at 2026-09-24 11:48:37 on dist-test-slave-dd32"
server_key: "3ff4af4f3c8619f088202b974f7fd8ea"
server_key_iv: "f0c1f17137abc14579eee1f575cff8bd"
server_key_version: "encryptionkey@0"
I20260924 11:48:37.934043 11720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "ef9889d5f89c4d128fe47a70bd39129f"
format_stamp: "Formatted at 2026-09-24 11:48:37 on dist-test-slave-dd32"
server_key: "3ff4af4f3c8619f088202b974f7fd8ea"
server_key_iv: "f0c1f17137abc14579eee1f575cff8bd"
server_key_version: "encryptionkey@0"
I20260924 11:48:37.942485 11720 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260924 11:48:37.947729 11734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:37.949549 11720 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 11:48:37.949750 11720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "ef9889d5f89c4d128fe47a70bd39129f"
format_stamp: "Formatted at 2026-09-24 11:48:37 on dist-test-slave-dd32"
server_key: "3ff4af4f3c8619f088202b974f7fd8ea"
server_key_iv: "f0c1f17137abc14579eee1f575cff8bd"
server_key_version: "encryptionkey@0"
I20260924 11:48:37.950024 11720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:37.957252 11720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:37.957970 11720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:37.958187 11720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:37.958983 11720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:37.960264 11720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:37.960346 11720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:37.960425 11720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:37.960479 11720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:37.991933 11720 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:33231
I20260924 11:48:37.991979 11846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:33231 every 8 connection(s)
I20260924 11:48:37.993269 11720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:37.994036 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11720
I20260924 11:48:37.994266 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:48:37.994967 1864 external_mini_cluster.cc:1428] Setting key 15de856516ac33daa20a01bd6555f2c0
I20260924 11:48:37.998056 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37425
--builtin_ntp_servers=127.1.210.20:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 11:48:38.001695 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:37.996000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:43155 (local address 127.1.210.62:37425)
0924 11:48:37.996456 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:37.996460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:37.997807 (+ 1347us) server_negotiation.cc:407] Connection header received
0924 11:48:37.999409 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:37.999415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:37.999476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:37.999606 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:38.000908 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:38.000914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:38.000919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:38.000936 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:38.000986 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:38.000999 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:38.001137 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:38.001350 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 11:48:38.001414 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":266,"threads_started":1}
I20260924 11:48:38.002571 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:37.996684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37425 (local address 127.1.210.1:43155)
0924 11:48:37.997662 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:37.997705 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:37.999163 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:37.999840 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:37.999849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:37.999938 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:38.000620 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:48:38.000623 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:38.000666 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:48:38.000746 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:38.001127 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:38.001137 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:38.001144 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:48:38.001416 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 11:48:38.001665 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":349,"threads_started":1}
I20260924 11:48:38.004065 11847 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37425
I20260924 11:48:38.004383 11847 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:38.005101 11847 heartbeater.cc:511] Master 127.1.210.62:37425 requested a full tablet report, sending...
I20260924 11:48:38.007279 11661 ts_manager.cc:194] Registered new tserver with Master: ef9889d5f89c4d128fe47a70bd39129f (127.1.210.1:33231)
W20260924 11:48:38.313195 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:38.313467 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:38.313555 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:38.320696 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:38.320792 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:38.320899 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:38.329743 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:38.330910 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:38.332567 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:38.345258 11857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:38.345235 11856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:38.345383 11859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:38.346141 11851 server_base.cc:1034] running on GCE node
I20260924 11:48:38.346668 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:38.347501 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:38.348696 11851 hybrid_clock.cc:648] HybridClock initialized: now 1790250518348648 us; error 35 us; skew 500 ppm
I20260924 11:48:38.349011 11851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:38.357262 11851 webserver.cc:460] Webserver started at http://127.1.210.2:42567/ using document root <none> and password file <none>
I20260924 11:48:38.357939 11851 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:38.358026 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:38.358353 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:38.361034 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "d1f1f1db3d3e4561b2f8b9879604139e"
format_stamp: "Formatted at 2026-09-24 11:48:38 on dist-test-slave-dd32"
server_key: "46ca5335f365fc014374c4df6ac15c2a"
server_key_iv: "004dbb388b8ab05f163ce5c0226160ea"
server_key_version: "encryptionkey@0"
I20260924 11:48:38.361934 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f1f1db3d3e4561b2f8b9879604139e"
format_stamp: "Formatted at 2026-09-24 11:48:38 on dist-test-slave-dd32"
server_key: "46ca5335f365fc014374c4df6ac15c2a"
server_key_iv: "004dbb388b8ab05f163ce5c0226160ea"
server_key_version: "encryptionkey@0"
I20260924 11:48:38.366818 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:48:38.370095 11865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:38.371333 11851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:48:38.371500 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "d1f1f1db3d3e4561b2f8b9879604139e"
format_stamp: "Formatted at 2026-09-24 11:48:38 on dist-test-slave-dd32"
server_key: "46ca5335f365fc014374c4df6ac15c2a"
server_key_iv: "004dbb388b8ab05f163ce5c0226160ea"
server_key_version: "encryptionkey@0"
I20260924 11:48:38.371711 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:38.377234 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:38.377955 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:38.378209 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:38.379155 11851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:38.380427 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:38.380546 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:38.380679 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:38.380766 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:38.426663 11851 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39607
I20260924 11:48:38.426728 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39607 every 8 connection(s)
I20260924 11:48:38.428516 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:38.429877 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11851
I20260924 11:48:38.430063 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:48:38.430589 1864 external_mini_cluster.cc:1428] Setting key 6ce0791fd94fd62b695eeef540eb7600
I20260924 11:48:38.433634 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37425
--builtin_ntp_servers=127.1.210.20:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 11:48:38.439009 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:38.431655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:47167 (local address 127.1.210.62:37425)
0924 11:48:38.431810 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:38.431816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:38.433688 (+ 1872us) server_negotiation.cc:407] Connection header received
0924 11:48:38.436353 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:38.436357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:38.436417 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:38.436499 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:38.438166 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:38.438172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:38.438177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:38.438195 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:38.438256 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:38.438271 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:38.438406 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:38.438663 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 11:48:38.438730 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 11:48:38.440325 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:38.432345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37425 (local address 127.1.210.2:47167)
0924 11:48:38.433525 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:38.433573 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:38.436061 (+ 2488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:38.436745 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:38.436758 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:38.436872 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:38.437757 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:48:38.437762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:38.437810 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:48:38.437965 (+ 155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:38.438388 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:38.438398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:38.438405 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:48:38.438740 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 11:48:38.438980 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
I20260924 11:48:38.441915 11978 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37425
I20260924 11:48:38.442203 11978 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:38.442821 11978 heartbeater.cc:511] Master 127.1.210.62:37425 requested a full tablet report, sending...
I20260924 11:48:38.444224 11661 ts_manager.cc:194] Registered new tserver with Master: d1f1f1db3d3e4561b2f8b9879604139e (127.1.210.2:39607)
W20260924 11:48:38.769223 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:38.769464 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:38.769623 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:38.776580 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:38.776672 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:38.776865 11981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:38.785679 11981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:38.786932 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:38.788621 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:38.801827 11987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:38.801841 11986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:38.801788 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:38.802774 11981 server_base.cc:1034] running on GCE node
I20260924 11:48:38.803311 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:38.804177 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:38.805362 11981 hybrid_clock.cc:648] HybridClock initialized: now 1790250518805303 us; error 57 us; skew 500 ppm
I20260924 11:48:38.805743 11981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:48:38.808050 11981 webserver.cc:460] Webserver started at http://127.1.210.3:44299/ using document root <none> and password file <none>
I20260924 11:48:38.808714 11981 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:38.808825 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:38.809204 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:38.811918 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "a5d3f6b0fa5743f996d36ca611dc672d"
format_stamp: "Formatted at 2026-09-24 11:48:38 on dist-test-slave-dd32"
server_key: "e6fe6b4ab9b45260134a73cf1cd788b1"
server_key_iv: "d080897357a21d3ec914d509f5c2ac5a"
server_key_version: "encryptionkey@0"
I20260924 11:48:38.812682 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "a5d3f6b0fa5743f996d36ca611dc672d"
format_stamp: "Formatted at 2026-09-24 11:48:38 on dist-test-slave-dd32"
server_key: "e6fe6b4ab9b45260134a73cf1cd788b1"
server_key_iv: "d080897357a21d3ec914d509f5c2ac5a"
server_key_version: "encryptionkey@0"
I20260924 11:48:38.817755 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:48:38.821264 11995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:38.822587 11981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:48:38.822896 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "a5d3f6b0fa5743f996d36ca611dc672d"
format_stamp: "Formatted at 2026-09-24 11:48:38 on dist-test-slave-dd32"
server_key: "e6fe6b4ab9b45260134a73cf1cd788b1"
server_key_iv: "d080897357a21d3ec914d509f5c2ac5a"
server_key_version: "encryptionkey@0"
I20260924 11:48:38.823165 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:38.829025 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:38.829878 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:38.830157 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:38.831103 11981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:38.832504 11981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:38.832640 11981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:38.832775 11981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:38.832881 11981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:38.866218 11981 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:42149
I20260924 11:48:38.866271 12107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:42149 every 8 connection(s)
I20260924 11:48:38.867625 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:38.871886 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 11981
I20260924 11:48:38.872087 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:48:38.872610 1864 external_mini_cluster.cc:1428] Setting key ccd44160939e784a396059e536fda29b
I20260924 11:48:38.876282 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:38.870196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:35649 (local address 127.1.210.62:37425)
0924 11:48:38.870373 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:38.870377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:38.871861 (+ 1484us) server_negotiation.cc:407] Connection header received
0924 11:48:38.874009 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:38.874015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:38.874080 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:38.874196 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:38.875547 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:38.875552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:38.875557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:38.875573 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:38.875616 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:38.875627 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:38.875741 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:38.875949 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 11:48:38.876021 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:48:38.877187 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:38.870694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37425 (local address 127.1.210.3:35649)
0924 11:48:38.871695 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:38.871747 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:38.873719 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:38.874437 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:38.874445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:38.874539 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:38.875206 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:48:38.875210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:38.875259 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:48:38.875378 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:38.875725 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:38.875734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:38.875740 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:48:38.876025 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 11:48:38.876264 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260924 11:48:38.878298 12108 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37425
I20260924 11:48:38.878578 12108 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:38.879115 12108 heartbeater.cc:511] Master 127.1.210.62:37425 requested a full tablet report, sending...
I20260924 11:48:38.880362 11661 ts_manager.cc:194] Registered new tserver with Master: a5d3f6b0fa5743f996d36ca611dc672d (127.1.210.3:42149)
I20260924 11:48:38.887753 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:48:38.893602 11849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:48:38.892052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.1.210.62:37425)
0924 11:48:38.892216 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:38.892225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:38.892616 (+ 391us) server_negotiation.cc:407] Connection header received
0924 11:48:38.892804 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:38.892810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:38.892871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:38.892969 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:38.893334 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40448: BlockingRecv error: recv got EOF from 127.0.0.1:40448 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:48:38.894259 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11720
I20260924 11:48:38.897624 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:38.895193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.1.210.1:33231)
0924 11:48:38.895715 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:38.895721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:38.895745 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:48:38.895796 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:38.895802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:38.895976 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:38.896089 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:38.896900 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:38.896906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:38.896917 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:38.896932 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:38.896968 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:38.896981 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:38.897130 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:38.897229 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 11:48:38.897334 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
I20260924 11:48:38.898731 11842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:38.984611 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11720
I20260924 11:48:39.000456 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11851
I20260924 11:48:39.003942 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:39.001384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.1.210.2:39607)
0924 11:48:39.001818 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:39.001831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:39.001855 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:48:39.001912 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:39.001918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:39.002120 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:39.002314 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:39.003263 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:39.003270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:39.003280 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:39.003297 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:39.003335 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:39.003348 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:39.003493 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:39.003596 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 11:48:39.003692 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":194,"threads_started":1}
I20260924 11:48:39.005182 11973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:39.084486 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11851
I20260924 11:48:39.099746 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11981
I20260924 11:48:39.103717 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:39.100801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.1.210.3:42149)
0924 11:48:39.101504 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:39.101511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:39.101540 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:48:39.101599 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:39.101604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:39.101832 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:39.101951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:39.102973 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:39.102979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:39.102990 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:48:39.103005 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:39.103042 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:48:39.103056 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:39.103184 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:39.103292 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 11:48:39.103410 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":409,"threads_started":1}
I20260924 11:48:39.105010 12103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:39.186079 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11981
I20260924 11:48:39.201360 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 11630
I20260924 11:48:39.202539 11691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:39.300932 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 11630
2026-09-24T11:48:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2294 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:48:39 dist-test-slave-dd32 krb5kdc[12126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:48:39 dist-test-slave-dd32 krb5kdc[12126](info): set up 2 sockets
Sep 24 11:48:39 dist-test-slave-dd32 krb5kdc[12126](info): commencing operation
krb5kdc: starting...
W20260924 11:48:40.406297 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:48:40 dist-test-slave-dd32 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:48:40.617681 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:34347
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:34347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:40.928182 12139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:40.928479 12139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:40.928651 12139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:40.939253 12139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:48:40.939388 12139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:40.939466 12139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:40.939548 12139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:48:40.939610 12139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:48:40.949319 12139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:34347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:34347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:40.950614 12139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:40.951946 12139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:40.963375 12144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:40.963451 12147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:40.963425 12145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:40.964177 12139 server_base.cc:1034] running on GCE node
I20260924 11:48:40.964829 12139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:40.966092 12139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:40.967283 12139 hybrid_clock.cc:648] HybridClock initialized: now 1790250520967242 us; error 39 us; skew 500 ppm
I20260924 11:48:40.967605 12139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:40 dist-test-slave-dd32 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:40.970533 12139 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:48:40.971918 12139 webserver.cc:460] Webserver started at http://127.1.210.62:46507/ using document root <none> and password file <none>
I20260924 11:48:40.972580 12139 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:40.972726 12139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:40.973012 12139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:40.975793 12139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "627378224870443bafc8ca418de76d49"
format_stamp: "Formatted at 2026-09-24 11:48:40 on dist-test-slave-dd32"
server_key: "c7131411aa8fc5c0c5d0b5e5244b6e02"
server_key_iv: "6d249299db056b2f1cb10fc21e479e82"
server_key_version: "encryptionkey@0"
I20260924 11:48:40.976524 12139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "627378224870443bafc8ca418de76d49"
format_stamp: "Formatted at 2026-09-24 11:48:40 on dist-test-slave-dd32"
server_key: "c7131411aa8fc5c0c5d0b5e5244b6e02"
server_key_iv: "6d249299db056b2f1cb10fc21e479e82"
server_key_version: "encryptionkey@0"
I20260924 11:48:40.981323 12139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260924 11:48:40.984412 12154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:40.985648 12139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 11:48:40.985826 12139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "627378224870443bafc8ca418de76d49"
format_stamp: "Formatted at 2026-09-24 11:48:40 on dist-test-slave-dd32"
server_key: "c7131411aa8fc5c0c5d0b5e5244b6e02"
server_key_iv: "6d249299db056b2f1cb10fc21e479e82"
server_key_version: "encryptionkey@0"
I20260924 11:48:40.986008 12139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:41.002758 12139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:41.006388 12139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:41.006649 12139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:41.024053 12139 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:34347
I20260924 11:48:41.024083 12205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:34347 every 8 connection(s)
I20260924 11:48:41.025316 12139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:48:41.027000 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 12139
I20260924 11:48:41.027230 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:48:41.028002 1864 external_mini_cluster.cc:1428] Setting key ed393e3b80a5efeaeffa9fcf0e614428
I20260924 11:48:41.029560 12206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:41.040254 12206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49: Bootstrap starting.
Sep 24 11:48:41 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:41.043823 12206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:41.044807 12206 log.cc:824] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:41.047781 12206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49: No bootstrap required, opened a new log
I20260924 11:48:41.048759 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:41.029977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.1.210.62:34347)
0924 11:48:41.030689 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:41.030699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:41.030739 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:48:41.032221 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:41.032240 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:41.032594 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:41.032987 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:41.034406 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:41.035674 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:41.036923 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:41.037685 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:41.037898 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:41.041991 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:41.042000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:41.042013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:41.042044 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:41.045040 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:41.045818 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:41.045830 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:41.045838 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:41.046039 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:41.046435 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:41.046438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:41.046440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:41.046875 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:41.047097 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:41.047503 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 11:48:41.047857 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":333,"threads_started":1}
I20260924 11:48:41.055305 12206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627378224870443bafc8ca418de76d49" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34347 } }
I20260924 11:48:41.055634 12206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:41.055745 12206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627378224870443bafc8ca418de76d49, State: Initialized, Role: FOLLOWER
I20260924 11:48:41.056231 12206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627378224870443bafc8ca418de76d49" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34347 } }
I20260924 11:48:41.056461 12206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:48:41.056599 12206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:48:41.056771 12206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:41.058808 12206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627378224870443bafc8ca418de76d49" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34347 } }
I20260924 11:48:41.059259 12206 leader_election.cc:302] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627378224870443bafc8ca418de76d49; no voters:
I20260924 11:48:41.059775 12206 leader_election.cc:288] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:48:41.059891 12211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:41.060628 12211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [term 1 LEADER]: Becoming Leader. State: Replica: 627378224870443bafc8ca418de76d49, State: Running, Role: LEADER
I20260924 11:48:41.061141 12211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627378224870443bafc8ca418de76d49" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34347 } }
I20260924 11:48:41.061793 12206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:48:41.063732 12213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627378224870443bafc8ca418de76d49. Latest consensus state: current_term: 1 leader_uuid: "627378224870443bafc8ca418de76d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627378224870443bafc8ca418de76d49" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34347 } } }
I20260924 11:48:41.063714 12212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627378224870443bafc8ca418de76d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627378224870443bafc8ca418de76d49" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34347 } } }
I20260924 11:48:41.064095 12213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:41.064222 12212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49 [sys.catalog]: This master's current role is: LEADER
I20260924 11:48:41.066459 12216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:48:41.071055 12216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:48:41.079322 12216 catalog_manager.cc:1347] Generated new cluster ID: 4e3cb8829dea4b51aeba28b98ff3617e
I20260924 11:48:41.079421 12216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:48:41.095680 12216 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:48:41.097479 12216 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:48:41.106096 12216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49: Generated new TSK 0
I20260924 11:48:41.107081 12216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:48:41.158367 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34347
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:41.492949 12232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:41.493335 12232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:41.493530 12232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:41.500847 12232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:41.500952 12232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:41.501139 12232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:48:41.511000 12232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:41.512177 12232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:41.513537 12232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:41.526815 12238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:41.526793 12237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:41.526970 12240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:41.527791 12232 server_base.cc:1034] running on GCE node
I20260924 11:48:41.528403 12232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:41.529309 12232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:41.530526 12232 hybrid_clock.cc:648] HybridClock initialized: now 1790250521530492 us; error 47 us; skew 500 ppm
I20260924 11:48:41.530840 12232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:41 dist-test-slave-dd32 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250521, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:41.534003 12232 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:48:41.535302 12232 webserver.cc:460] Webserver started at http://127.1.210.1:38491/ using document root <none> and password file <none>
I20260924 11:48:41.535941 12232 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:41.536038 12232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:41.536271 12232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:41.539119 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "ab893e1f86224d16ad0085a1a45441fa"
format_stamp: "Formatted at 2026-09-24 11:48:41 on dist-test-slave-dd32"
server_key: "4e39c7e590a61d8e7f63bdbdb9f8bfab"
server_key_iv: "33f5b3fdddf4cf409a3cf3ee5a23d6c4"
server_key_version: "encryptionkey@0"
I20260924 11:48:41.539827 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "ab893e1f86224d16ad0085a1a45441fa"
format_stamp: "Formatted at 2026-09-24 11:48:41 on dist-test-slave-dd32"
server_key: "4e39c7e590a61d8e7f63bdbdb9f8bfab"
server_key_iv: "33f5b3fdddf4cf409a3cf3ee5a23d6c4"
server_key_version: "encryptionkey@0"
I20260924 11:48:41.544612 12232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 11:48:41.548180 12247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:41.549660 12232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:41.549850 12232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "ab893e1f86224d16ad0085a1a45441fa"
format_stamp: "Formatted at 2026-09-24 11:48:41 on dist-test-slave-dd32"
server_key: "4e39c7e590a61d8e7f63bdbdb9f8bfab"
server_key_iv: "33f5b3fdddf4cf409a3cf3ee5a23d6c4"
server_key_version: "encryptionkey@0"
I20260924 11:48:41.550050 12232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:41.562057 12232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:41.565996 12232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:41.566279 12232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:41.567279 12232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:41.568645 12232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:41.568755 12232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:41.568898 12232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:41.568969 12232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:41.600004 12232 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:39429
I20260924 11:48:41.600045 12359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:39429 every 8 connection(s)
I20260924 11:48:41.601338 12232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:48:41.609768 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 12232
I20260924 11:48:41.609977 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:48:41.610618 1864 external_mini_cluster.cc:1428] Setting key 6413edcfba8c37a45549979793d29581
Sep 24 11:48:41 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250521, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:41.620111 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:41.603775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:36197 (local address 127.1.210.62:34347)
0924 11:48:41.604223 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:41.604230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:41.605220 (+ 990us) server_negotiation.cc:407] Connection header received
0924 11:48:41.607093 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:41.607097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:41.607158 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:41.607257 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:41.608613 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:41.609369 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:41.611032 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:41.611413 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:41.611550 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:41.614858 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:41.614863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:41.614866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:41.614893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:41.617279 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:41.617961 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:41.617983 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:41.617986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:41.618095 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:41.618521 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:41.618541 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:41.618544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:41.618808 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:41.618962 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:41.619563 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 11:48:41.619761 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":207,"threads_started":1}
I20260924 11:48:41.620993 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:41.604327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:34347 (local address 127.1.210.1:36197)
0924 11:48:41.605016 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:41.605055 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:41.606855 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:41.607446 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:41.607454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:41.607924 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:41.608437 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:41.608452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:41.609521 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:41.609525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:41.610885 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:41.610900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:41.611631 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:41.611640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:41.611904 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:41.612574 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:41.612595 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:41.614650 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:41.617404 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:41.617411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:41.617425 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:41.617833 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:41.618200 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:41.618204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:41.618207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:41.618411 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:41.618951 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:41.618958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:41.619238 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 11:48:41.619478 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 11:48:41.619950 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":261,"threads_started":1}
I20260924 11:48:41.622534 12360 heartbeater.cc:347] Connected to a master server at 127.1.210.62:34347
I20260924 11:48:41.622853 12360 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:41.623540 12360 heartbeater.cc:511] Master 127.1.210.62:34347 requested a full tablet report, sending...
I20260924 11:48:41.625720 12171 ts_manager.cc:194] Registered new tserver with Master: ab893e1f86224d16ad0085a1a45441fa (127.1.210.1:39429)
I20260924 11:48:41.627873 12171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:36197
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:48:41.658099 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34347
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:41.969873 12366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:41.970124 12366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:41.970322 12366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:41.977330 12366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:41.977447 12366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:41.977633 12366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:48:41.986756 12366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:41.987959 12366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:41.989255 12366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:42.001843 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:42.001891 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:42.001813 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:42.003055 12366 server_base.cc:1034] running on GCE node
I20260924 11:48:42.003618 12366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:42.004455 12366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:42.005674 12366 hybrid_clock.cc:648] HybridClock initialized: now 1790250522005639 us; error 48 us; skew 500 ppm
I20260924 11:48:42.005971 12366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:42 dist-test-slave-dd32 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:42.008855 12366 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:48:42.010346 12366 webserver.cc:460] Webserver started at http://127.1.210.2:45599/ using document root <none> and password file <none>
I20260924 11:48:42.010980 12366 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:42.011121 12366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:42.011408 12366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:42.014194 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "575c9a495982401fae8e02e036415963"
format_stamp: "Formatted at 2026-09-24 11:48:42 on dist-test-slave-dd32"
server_key: "947b3d277d772d68bdef2841fdfbdaf0"
server_key_iv: "a48b70f0fb718d77371cd99c503b4d5e"
server_key_version: "encryptionkey@0"
I20260924 11:48:42.015194 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "575c9a495982401fae8e02e036415963"
format_stamp: "Formatted at 2026-09-24 11:48:42 on dist-test-slave-dd32"
server_key: "947b3d277d772d68bdef2841fdfbdaf0"
server_key_iv: "a48b70f0fb718d77371cd99c503b4d5e"
server_key_version: "encryptionkey@0"
I20260924 11:48:42.020056 12366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:48:42.023249 12381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:42.024761 12366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:48:42.024909 12366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "575c9a495982401fae8e02e036415963"
format_stamp: "Formatted at 2026-09-24 11:48:42 on dist-test-slave-dd32"
server_key: "947b3d277d772d68bdef2841fdfbdaf0"
server_key_iv: "a48b70f0fb718d77371cd99c503b4d5e"
server_key_version: "encryptionkey@0"
I20260924 11:48:42.025125 12366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:42.046660 12366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:42.050257 12366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:42.050474 12366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:42.051268 12366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:42.052512 12366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:42.052582 12366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:42.052662 12366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:42.052735 12366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:42.084854 12366 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:33127
I20260924 11:48:42.084896 12493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:33127 every 8 connection(s)
I20260924 11:48:42.086071 12366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:48:42.088092 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 12366
I20260924 11:48:42.088282 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:48:42.088868 1864 external_mini_cluster.cc:1428] Setting key be51170d575d074297c5026bd7d1f0da
Sep 24 11:48:42 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:42.105832 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.088700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:60233 (local address 127.1.210.62:34347)
0924 11:48:42.088912 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.088920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.090249 (+ 1329us) server_negotiation.cc:407] Connection header received
0924 11:48:42.092078 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.092083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.092164 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.092277 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:42.093771 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.094490 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.095999 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.096410 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.096544 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.100402 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:42.100407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:42.100410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:42.100439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:42.102940 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.103619 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.103624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.103634 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.103750 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.104169 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.104174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.104177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.104431 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:42.104584 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.105256 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.105468 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:48:42.106820 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.089331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:34347 (local address 127.1.210.2:60233)
0924 11:48:42.090102 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.090144 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.091827 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.092503 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.092511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.092986 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:42.093597 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.093614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.094631 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.094643 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.095855 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.095867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.096631 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.096639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.096865 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.097567 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:42.097593 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:42.100171 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:42.103077 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:42.103085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:42.103099 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:42.103490 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:42.103867 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:42.103871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:42.103875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:42.104052 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:42.104557 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:42.104565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:42.104873 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.105164 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.105639 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":318,"threads_started":1}
I20260924 11:48:42.108229 12494 heartbeater.cc:347] Connected to a master server at 127.1.210.62:34347
I20260924 11:48:42.108577 12494 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:42.109506 12494 heartbeater.cc:511] Master 127.1.210.62:34347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:48:42.111014 12171 ts_manager.cc:194] Registered new tserver with Master: 575c9a495982401fae8e02e036415963 (127.1.210.2:33127)
I20260924 11:48:42.111773 12171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:60233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:48:42.136122 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34347
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:48:42.450286 12499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:48:42.450603 12499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:48:42.450768 12499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:48:42.458220 12499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:48:42.458360 12499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:48:42.458523 12499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:48:42.468144 12499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:48:42.469478 12499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:48:42.470770 12499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:48:42.483433 12505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:42.483476 12507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:48:42.483563 12504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:48:42.484184 12499 server_base.cc:1034] running on GCE node
I20260924 11:48:42.484746 12499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:48:42.485643 12499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:48:42.486826 12499 hybrid_clock.cc:648] HybridClock initialized: now 1790250522486768 us; error 52 us; skew 500 ppm
I20260924 11:48:42.487149 12499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:48:42 dist-test-slave-dd32 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:48:42.490331 12499 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:48:42.491577 12499 webserver.cc:460] Webserver started at http://127.1.210.3:46571/ using document root <none> and password file <none>
I20260924 11:48:42.492152 12499 fs_manager.cc:359] Metadata directory not provided
I20260924 11:48:42.492251 12499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:48:42.492471 12499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:48:42.495237 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44"
format_stamp: "Formatted at 2026-09-24 11:48:42 on dist-test-slave-dd32"
server_key: "55fc9c5865bd8673345a7448b948774d"
server_key_iv: "bb7f946b9cbef1bef99a7cafbae690e7"
server_key_version: "encryptionkey@0"
I20260924 11:48:42.495965 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44"
format_stamp: "Formatted at 2026-09-24 11:48:42 on dist-test-slave-dd32"
server_key: "55fc9c5865bd8673345a7448b948774d"
server_key_iv: "bb7f946b9cbef1bef99a7cafbae690e7"
server_key_version: "encryptionkey@0"
I20260924 11:48:42.500813 12499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 11:48:42.504123 12514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:42.505605 12499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:48:42.505789 12499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44"
format_stamp: "Formatted at 2026-09-24 11:48:42 on dist-test-slave-dd32"
server_key: "55fc9c5865bd8673345a7448b948774d"
server_key_iv: "bb7f946b9cbef1bef99a7cafbae690e7"
server_key_version: "encryptionkey@0"
I20260924 11:48:42.506014 12499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:48:42.518965 12499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:48:42.522718 12499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:48:42.522991 12499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:48:42.523890 12499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:48:42.525210 12499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:48:42.525315 12499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:42.525465 12499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:48:42.525542 12499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:48:42.557827 12499 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:42219
I20260924 11:48:42.557873 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:42219 every 8 connection(s)
I20260924 11:48:42.559139 12499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:48:42.565907 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 12499
I20260924 11:48:42.566085 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:48:42.566524 1864 external_mini_cluster.cc:1428] Setting key 7fd6b6724f97ac591e705e6293625d67
Sep 24 11:48:42 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:48:42.578725 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.561757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:47437 (local address 127.1.210.62:34347)
0924 11:48:42.561921 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.561925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.563172 (+ 1247us) server_negotiation.cc:407] Connection header received
0924 11:48:42.564924 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.564930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.565056 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.565196 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:42.566763 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.567563 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.569539 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.569931 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.570022 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.573423 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:42.573429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:42.573432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:42.573462 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:42.575781 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.576497 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.576502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.576506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.576623 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.577049 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.577056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.577081 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.577354 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:42.577502 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.578146 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.578373 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:48:42.579831 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.562211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:34347 (local address 127.1.210.3:47437)
0924 11:48:42.563005 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.563041 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.564665 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.565428 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.565436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.565903 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:48:42.566568 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.566610 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.567840 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.567846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.569437 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.569450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.570099 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.570104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.570294 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.571047 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:48:42.571095 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:48:42.573232 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:48:42.575933 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:42.575941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:42.575957 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:42.576373 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:42.576728 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:48:42.576735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:48:42.576741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:48:42.576939 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:48:42.577477 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:48:42.577485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:48:42.577807 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.578061 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.578595 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":291,"threads_started":1}
I20260924 11:48:42.581248 12627 heartbeater.cc:347] Connected to a master server at 127.1.210.62:34347
I20260924 11:48:42.581559 12627 heartbeater.cc:464] Registering TS with master...
I20260924 11:48:42.582346 12627 heartbeater.cc:511] Master 127.1.210.62:34347 requested a full tablet report, sending...
I20260924 11:48:42.583612 12171 ts_manager.cc:194] Registered new tserver with Master: d7b9b1ae1bc54d4a9ceca9795a36cd44 (127.1.210.3:42219)
I20260924 11:48:42.584333 12171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:47437
I20260924 11:48:42.593962 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:48:42.610639 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.598285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.1.210.62:34347)
0924 11:48:42.598456 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.598460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.598929 (+ 469us) server_negotiation.cc:407] Connection header received
0924 11:48:42.599057 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.599060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.599116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.599192 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:42.600514 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.601652 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.602857 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.603489 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.603601 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.605880 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:42.605885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:42.605888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:42.605917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:42.608146 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.608710 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.608713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.608723 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.608837 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.609278 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.609285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.609288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.609575 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:42.609848 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.610147 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.610345 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:48:42.614516 12171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:48:42.621052 12171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:48:42.630581 12360 heartbeater.cc:503] Master 127.1.210.62:34347 was elected leader, sending a full tablet report...
I20260924 11:48:42.647456 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.632731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43912)
0924 11:48:42.633392 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.633436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.633594 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.634330 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.634341 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.634378 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:42.634846 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.634863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.638116 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.638125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.642445 (+ 4320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.642465 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.645896 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.645906 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.646129 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.646739 (+ 610us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.646874 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.647036 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":501,"thread_start_us":1214,"threads_started":1}
I20260924 11:48:42.647832 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.632914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.1.210.1:39429)
0924 11:48:42.633871 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.633877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.633898 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:48:42.633961 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.633966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.634133 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.634325 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:42.635092 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.637933 (+ 2841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.642703 (+ 4770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.645650 (+ 2947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.646241 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.646488 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.647353 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.647513 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":665,"threads_started":1}
I20260924 11:48:42.651263 12295 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb61ab2fbfc4bd090f85b9112584c13 (DEFAULT_TABLE table=test-table [id=fdaa165ae4d7423d8aa0c0833f0fc944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:42.652738 12295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb61ab2fbfc4bd090f85b9112584c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:42.652792 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.633808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:42219 (local address 127.0.0.1:58638)
0924 11:48:42.634283 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.634299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.634405 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.645804 (+ 11399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.645813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.645850 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:42.646457 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.646472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.648897 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.648904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.650276 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.650287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.651965 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.651972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.652158 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.652192 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.652296 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.652419 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":24,"thread_start_us":696,"threads_started":1}
I20260924 11:48:42.652837 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.633489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.1.210.3:42219)
0924 11:48:42.645252 (+ 11763us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.645261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.645291 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:48:42.645380 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.645388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.645616 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.645808 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:42.646696 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.648739 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.650435 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.651756 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.651890 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.652096 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.652363 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.652503 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11593,"thread_start_us":12517,"threads_started":1}
I20260924 11:48:42.656267 12562 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb61ab2fbfc4bd090f85b9112584c13 (DEFAULT_TABLE table=test-table [id=fdaa165ae4d7423d8aa0c0833f0fc944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:42.657668 12562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb61ab2fbfc4bd090f85b9112584c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:42.658123 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.634846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:33127 (local address 127.0.0.1:54192)
0924 11:48:42.647021 (+ 12175us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.647038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.647171 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.647959 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.647966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.647989 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:42.648347 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.648358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.653610 (+ 5252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.653615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.655546 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.655561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.657310 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.657316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.657483 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.657518 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.657617 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.657819 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12035,"thread_start_us":11892,"threads_started":1}
I20260924 11:48:42.658126 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.635056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.1.210.2:33127)
0924 11:48:42.646551 (+ 11495us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.646558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.647155 (+ 597us) server_negotiation.cc:407] Connection header received
0924 11:48:42.647296 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.647303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.647536 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.647708 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:42.648504 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.653463 (+ 4959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.655719 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.657122 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.657271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.657455 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.657670 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.657820 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11332,"thread_start_us":13667,"threads_started":1}
I20260924 11:48:42.661761 12429 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb61ab2fbfc4bd090f85b9112584c13 (DEFAULT_TABLE table=test-table [id=fdaa165ae4d7423d8aa0c0833f0fc944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:48:42.663030 12429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb61ab2fbfc4bd090f85b9112584c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:48:42.665307 12645 tablet_bootstrap.cc:492] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Bootstrap starting.
I20260924 11:48:42.668339 12645 tablet_bootstrap.cc:654] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:42.669363 12645 log.cc:824] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:42.673038 12645 tablet_bootstrap.cc:492] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: No bootstrap required, opened a new log
I20260924 11:48:42.673170 12647 tablet_bootstrap.cc:492] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Bootstrap starting.
I20260924 11:48:42.673318 12645 ts_tablet_manager.cc:1397] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260924 11:48:42.676501 12647 tablet_bootstrap.cc:654] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:42.676731 12648 tablet_bootstrap.cc:492] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Bootstrap starting.
I20260924 11:48:42.677681 12647 log.cc:824] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:42.680020 12647 tablet_bootstrap.cc:492] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: No bootstrap required, opened a new log
I20260924 11:48:42.680317 12647 ts_tablet_manager.cc:1397] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260924 11:48:42.680826 12648 tablet_bootstrap.cc:654] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Neither blocks nor log segments found. Creating new log.
I20260924 11:48:42.682001 12648 log.cc:824] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Log is configured to *not* fsync() on all Append() calls
I20260924 11:48:42.682507 12645 raft_consensus.cc:348] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.682934 12645 raft_consensus.cc:374] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:42.683033 12645 raft_consensus.cc:729] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab893e1f86224d16ad0085a1a45441fa, State: Initialized, Role: FOLLOWER
I20260924 11:48:42.683619 12645 consensus_queue.cc:260] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.684813 12648 tablet_bootstrap.cc:492] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: No bootstrap required, opened a new log
I20260924 11:48:42.685127 12648 ts_tablet_manager.cc:1397] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:48:42.685357 12645 ts_tablet_manager.cc:1428] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 11:48:42.688582 12647 raft_consensus.cc:348] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.688989 12647 raft_consensus.cc:374] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:42.689110 12647 raft_consensus.cc:729] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b9b1ae1bc54d4a9ceca9795a36cd44, State: Initialized, Role: FOLLOWER
I20260924 11:48:42.689685 12647 consensus_queue.cc:260] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.690743 12627 heartbeater.cc:503] Master 127.1.210.62:34347 was elected leader, sending a full tablet report...
I20260924 11:48:42.691246 12647 ts_tablet_manager.cc:1428] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260924 11:48:42.694365 12648 raft_consensus.cc:348] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.694772 12648 raft_consensus.cc:374] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:48:42.694861 12648 raft_consensus.cc:729] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575c9a495982401fae8e02e036415963, State: Initialized, Role: FOLLOWER
I20260924 11:48:42.695350 12648 consensus_queue.cc:260] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.696298 12494 heartbeater.cc:503] Master 127.1.210.62:34347 was elected leader, sending a full tablet report...
I20260924 11:48:42.696669 12648 ts_tablet_manager.cc:1428] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
W20260924 11:48:42.810827 12628 tablet.cc:2369] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:48:42.812052 12651 raft_consensus.cc:482] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:48:42.812330 12651 raft_consensus.cc:504] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.814005 12651 leader_election.cc:288] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b9b1ae1bc54d4a9ceca9795a36cd44 (127.1.210.3:42219), 575c9a495982401fae8e02e036415963 (127.1.210.2:33127)
I20260924 11:48:42.821398 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.814476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:59013 (local address 127.1.210.3:42219)
0924 11:48:42.814641 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.814646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.815246 (+ 600us) server_negotiation.cc:407] Connection header received
0924 11:48:42.815490 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.815495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.815576 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.815696 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:42.816483 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.817449 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.819455 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.820466 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.820610 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.820711 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.821002 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.821163 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:48:42.821533 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.815277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:33127 (local address 127.1.210.1:51011)
0924 11:48:42.815918 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.815931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.816041 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.816352 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.816355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.816371 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:42.816778 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.816791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.817898 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.817902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.819441 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.819455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.820800 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.820806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.820945 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.820966 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.821063 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.821211 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":362,"threads_started":1}
I20260924 11:48:42.821708 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.815475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:51011 (local address 127.1.210.2:33127)
0924 11:48:42.815692 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.815697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.816019 (+ 322us) server_negotiation.cc:407] Connection header received
0924 11:48:42.816154 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.816159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.816233 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.816334 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:48:42.816936 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.817765 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.819593 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.820564 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.820726 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.820824 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.821140 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.821265 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:48:42.821724 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.814434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:42219 (local address 127.1.210.1:59013)
0924 11:48:42.815145 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:42.815166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:48:42.815317 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:48:42.815847 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:48:42.815851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:48:42.815871 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:48:42.816328 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.816340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.817571 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.817576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.819278 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:48:42.819304 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.820659 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.820665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:48:42.820839 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.820862 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:48:42.820964 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:48:42.821115 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":4,"spinlock_wait_cycles":24320,"thread_start_us":382,"threads_started":1}
I20260924 11:48:42.822549 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb61ab2fbfc4bd090f85b9112584c13" candidate_uuid: "ab893e1f86224d16ad0085a1a45441fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" is_pre_election: true
I20260924 11:48:42.822546 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb61ab2fbfc4bd090f85b9112584c13" candidate_uuid: "ab893e1f86224d16ad0085a1a45441fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575c9a495982401fae8e02e036415963" is_pre_election: true
I20260924 11:48:42.823004 12582 raft_consensus.cc:2454] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab893e1f86224d16ad0085a1a45441fa in term 0.
I20260924 11:48:42.823007 12449 raft_consensus.cc:2454] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab893e1f86224d16ad0085a1a45441fa in term 0.
I20260924 11:48:42.823606 12248 leader_election.cc:302] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575c9a495982401fae8e02e036415963, ab893e1f86224d16ad0085a1a45441fa; no voters:
I20260924 11:48:42.823961 12651 raft_consensus.cc:2781] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:48:42.824141 12651 raft_consensus.cc:482] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:48:42.824244 12651 raft_consensus.cc:3037] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:42.826470 12651 raft_consensus.cc:504] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.827018 12651 leader_election.cc:288] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [CANDIDATE]: Term 1 election: Requested vote from peers d7b9b1ae1bc54d4a9ceca9795a36cd44 (127.1.210.3:42219), 575c9a495982401fae8e02e036415963 (127.1.210.2:33127)
I20260924 11:48:42.827435 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb61ab2fbfc4bd090f85b9112584c13" candidate_uuid: "ab893e1f86224d16ad0085a1a45441fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44"
I20260924 11:48:42.827518 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb61ab2fbfc4bd090f85b9112584c13" candidate_uuid: "ab893e1f86224d16ad0085a1a45441fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575c9a495982401fae8e02e036415963"
I20260924 11:48:42.827610 12582 raft_consensus.cc:3037] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:42.827754 12449 raft_consensus.cc:3037] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:48:42.829799 12582 raft_consensus.cc:2454] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab893e1f86224d16ad0085a1a45441fa in term 1.
I20260924 11:48:42.830164 12449 raft_consensus.cc:2454] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab893e1f86224d16ad0085a1a45441fa in term 1.
I20260924 11:48:42.830185 12248 leader_election.cc:302] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab893e1f86224d16ad0085a1a45441fa, d7b9b1ae1bc54d4a9ceca9795a36cd44; no voters:
I20260924 11:48:42.830523 12651 raft_consensus.cc:2781] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:48:42.830940 12651 raft_consensus.cc:686] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 1 LEADER]: Becoming Leader. State: Replica: ab893e1f86224d16ad0085a1a45441fa, State: Running, Role: LEADER
I20260924 11:48:42.831390 12651 consensus_queue.cc:237] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } }
I20260924 11:48:42.835485 12171 catalog_manager.cc:5495] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa reported cstate change: term changed from 0 to 1, leader changed from <none> to ab893e1f86224d16ad0085a1a45441fa (127.1.210.1). New cstate: current_term: 1 leader_uuid: "ab893e1f86224d16ad0085a1a45441fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab893e1f86224d16ad0085a1a45441fa" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 39429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 } health_report { overall_health: UNKNOWN } } }
W20260924 11:48:42.837914 12495 tablet.cc:2369] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:48:42.853637 12361 tablet.cc:2369] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:48:42.877290 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.872245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.1.210.1:39429)
0924 11:48:42.872464 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.872468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.872482 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:48:42.872558 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.872562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.872642 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.872795 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:48:42.873212 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.873920 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.875240 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.875636 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.875762 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.876226 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:48:42.876716 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:48:42.876876 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.876931 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.877002 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:48:42.884557 12449 raft_consensus.cc:1261] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 1 FOLLOWER]: Refusing update from remote peer ab893e1f86224d16ad0085a1a45441fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:42.884678 12582 raft_consensus.cc:1261] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 1 FOLLOWER]: Refusing update from remote peer ab893e1f86224d16ad0085a1a45441fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:48:42.885514 12656 consensus_queue.cc:1035] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "575c9a495982401fae8e02e036415963" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:42.885870 12651 consensus_queue.cc:1035] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b9b1ae1bc54d4a9ceca9795a36cd44" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:48:42.899508 12660 mvcc.cc:205] Tried to move back new op lower bound from 7332866141726781440 to 7332866141520834560. Current Snapshot: MvccSnapshot[applied={T|T < 7332866141726781440}]
I20260924 11:48:42.901722 12661 mvcc.cc:205] Tried to move back new op lower bound from 7332866141726781440 to 7332866141520834560. Current Snapshot: MvccSnapshot[applied={T|T < 7332866141726781440}]
I20260924 11:48:42.912303 12170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:42.912554 12170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:48:42.916242 12170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 627378224870443bafc8ca418de76d49: Sending DeleteTablet for 3 replicas of tablet 2bb61ab2fbfc4bd090f85b9112584c13
I20260924 11:48:42.917327 12295 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb61ab2fbfc4bd090f85b9112584c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:42 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:43912
I20260924 11:48:42.917520 12429 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb61ab2fbfc4bd090f85b9112584c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:42 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:54192
I20260924 11:48:42.918377 12562 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb61ab2fbfc4bd090f85b9112584c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:48:42 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:58638
I20260924 11:48:42.918844 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 12232
I20260924 11:48:42.942830 12671 tablet_replica.cc:330] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: stopping tablet replica
I20260924 11:48:42.943415 12671 raft_consensus.cc:2229] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:42.944058 12670 tablet_replica.cc:330] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: stopping tablet replica
I20260924 11:48:42.944602 12671 raft_consensus.cc:2258] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:42.945113 12670 raft_consensus.cc:2229] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:48:42.945937 12670 raft_consensus.cc:2258] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:42.947960 12671 ts_tablet_manager.cc:1905] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:42.954849 12669 tablet_replica.cc:330] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: stopping tablet replica
I20260924 11:48:42.955257 12669 raft_consensus.cc:2229] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:48:42.955287 12670 ts_tablet_manager.cc:1905] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:42.956004 12669 raft_consensus.cc:2258] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:48:42.956187 12671 ts_tablet_manager.cc:1918] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:42.956382 12671 log.cc:1196] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/2bb61ab2fbfc4bd090f85b9112584c13
I20260924 11:48:42.956964 12671 ts_tablet_manager.cc:1939] T 2bb61ab2fbfc4bd090f85b9112584c13 P d7b9b1ae1bc54d4a9ceca9795a36cd44: Deleting consensus metadata
Sep 24 11:48:42 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:48:42.958914 12155 catalog_manager.cc:4841] TS d7b9b1ae1bc54d4a9ceca9795a36cd44 (127.1.210.3:42219): tablet 2bb61ab2fbfc4bd090f85b9112584c13 (table test-table [id=fdaa165ae4d7423d8aa0c0833f0fc944]) successfully deleted
I20260924 11:48:42.960959 12669 ts_tablet_manager.cc:1905] T 2bb61ab2fbfc4bd090f85b9112584c13 P ab893e1f86224d16ad0085a1a45441fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:48:42.962554 12670 ts_tablet_manager.cc:1918] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:48:42.962713 12670 log.cc:1196] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/2bb61ab2fbfc4bd090f85b9112584c13
I20260924 11:48:42.963220 12670 ts_tablet_manager.cc:1939] T 2bb61ab2fbfc4bd090f85b9112584c13 P 575c9a495982401fae8e02e036415963: Deleting consensus metadata
I20260924 11:48:42.964865 12155 catalog_manager.cc:4841] TS 575c9a495982401fae8e02e036415963 (127.1.210.2:33127): tablet 2bb61ab2fbfc4bd090f85b9112584c13 (table test-table [id=fdaa165ae4d7423d8aa0c0833f0fc944]) successfully deleted
I20260924 11:48:42.967214 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:42.920626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.1.210.1:39429)
0924 11:48:42.920856 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:42.920864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:42.940735 (+ 19871us) server_negotiation.cc:407] Connection header received
0924 11:48:42.940936 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:42.940946 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:42.941103 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:42.941237 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:42.943546 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.945242 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.947813 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:42.948772 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:42.948897 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:42.959637 (+ 10740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:42.959650 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:42.959668 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:42.959716 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:42.963582 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.964513 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.964522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.964526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.964696 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:42.965215 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:42.965220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:42.965223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:42.965658 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:42.965966 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:42.966564 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 11:48:42.966806 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:48:42.967935 12355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:43.065224 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 12232
I20260924 11:48:43.083695 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 12366
W20260924 11:48:43.083716 12156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.1:39429 (error 108)
I20260924 11:48:43.085713 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.085226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43936)
0924 11:48:43.085408 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.085537 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 11:48:43.086055 12156 catalog_manager.cc:4584] TS ab893e1f86224d16ad0085a1a45441fa (127.1.210.1:39429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bb61ab2fbfc4bd090f85b9112584c13: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Sep 24 11:48:43 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:48:43.099421 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.084785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.1.210.2:33127)
0924 11:48:43.084967 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:43.084974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:43.085041 (+ 67us) server_negotiation.cc:407] Connection header received
0924 11:48:43.085307 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:43.085314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:43.085393 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:43.085499 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:43.087122 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:43.087877 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:43.089248 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:43.089729 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:43.089869 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:43.093768 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:43.093782 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:43.093800 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:43.093834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:43.096732 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:43.097429 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:43.097436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:43.097439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:43.097548 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:43.097923 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:43.097932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:43.097934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:43.098337 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:43.098623 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:43.098914 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 11:48:43.099125 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:48:43.100041 12489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:43.139778 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.139226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43940)
0924 11:48:43.139463 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.139583 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 11:48:43.140851 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.140453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43946)
0924 11:48:43.140622 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.140702 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:48:43.189689 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 12366
I20260924 11:48:43.201558 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.201050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43962)
0924 11:48:43.201329 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.201417 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 11:48:43.202539 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.202211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43968)
0924 11:48:43.202380 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.202443 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:48:43.207520 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 12499
Sep 24 11:48:43 dist-test-slave-dd32 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:48:43.221416 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.208248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.1.210.3:42219)
0924 11:48:43.208430 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:48:43.208436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:48:43.208455 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:48:43.208607 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:48:43.208618 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:48:43.208681 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:48:43.208804 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:48:43.210149 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:43.210861 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:43.212049 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:48:43.212441 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:48:43.212570 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:48:43.216573 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:48:43.216586 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:48:43.216601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:48:43.216636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:48:43.219300 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:43.219830 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:43.219836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:43.219838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:43.219967 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:48:43.220266 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:48:43.220269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:48:43.220271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:48:43.220570 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:48:43.220787 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:48:43.220966 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 11:48:43.221177 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:48:43.221989 12622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:43.283021 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.282519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43982)
0924 11:48:43.282754 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.282849 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 11:48:43.284083 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:48:43.283686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:39429 (local address 127.0.0.1:43994)
0924 11:48:43.283907 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 11:48:43.283985 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 11:48:43.310109 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 12499
I20260924 11:48:43.326043 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 12139
I20260924 11:48:43.327420 12201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:48:43.434130 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 12139
2026-09-24T11:48:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4134 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:48:43 dist-test-slave-dd32 krb5kdc[12680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:48:43 dist-test-slave-dd32 krb5kdc[12680](info): set up 2 sockets
Sep 24 11:48:43 dist-test-slave-dd32 krb5kdc[12680](info): commencing operation
krb5kdc: starting...
W20260924 11:48:44.530009 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:48:44 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:48:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/rangeradmin_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/rangeradmin_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/rangerlookup_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/rangerlookup_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
I20260924 11:48:44.830088 1864 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1DchkY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/postgres -l logfile start
2026-09-24 11:48:47.259 UTC [12710] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 11:48:47.259 UTC [12710] LOG: listening on IPv4 address "127.1.210.20", port 55461
2026-09-24 11:48:47.263 UTC [12710] LOG: listening on Unix socket "/tmp/.s.PGSQL.55461"
2026-09-24 11:48:47.286 UTC [12713] LOG: database system was shut down at 2026-09-24 11:48:45 UTC
2026-09-24 11:48:47.290 UTC [12710] LOG: database system is ready to accept connections
I20260924 11:48:48.261641 1864 mini_postgres.cc:96] Postgres bound to 55461
2026-09-24 11:48:48.278 UTC [12721] FATAL: database "slave" does not exist
127.1.210.20:55461 - accepting connections
I20260924 11:48:48.279260 1864 mini_ranger.cc:161] Starting Ranger...
I20260924 11:48:48.302994 1864 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 11:48:48.742612 1864 mini_ranger.cc:87] Created ranger Postgres database
I20260924 11:48:48.742749 1864 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-admin
2026-09-24 11:48:49,010 [I] DB FLAVOR :POSTGRES
2026-09-24 11:48:49,010 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:48:49,010 [I] Checking connection..
2026-09-24 11:48:49,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 11:48:49,265 [I] Checking connection passed.
2026-09-24 11:48:49,265 [I] --------- Verifying version history table ---------
2026-09-24 11:48:49,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:48:49,521 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:48:49,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:48:49,766 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:48:49,766 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 11:48:49,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 11:48:50.009 UTC [12785] WARNING: there is no transaction in progress
2026-09-24 11:48:50,015 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 11:48:50,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:48:50,266 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 11:48:50,266 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:48:50,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 11:48:50,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 11:48:50,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dd32.c.gcp-upstream.internal','N') ;"
2026-09-24 11:48:50,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:48:51,233 [I] Table x_portal_user does not exist in database ranger
2026-09-24 11:48:51,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:48:51,492 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 11:48:51,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:48:51,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:48:51,962 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 11:48:51,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 11:48:52.879 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:52.895 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:52.925 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:52.941 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:52.960 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:52.980 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.033 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.039 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.052 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.062 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.071 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.084 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.093 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.099 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.105 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53.111 UTC [12893] WARNING: there is no transaction in progress
2026-09-24 11:48:53,718 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 11:48:53,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 11:48:53,941 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 11:48:53,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:48:54,191 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 11:48:54,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:48:54,431 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 11:48:54,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:48:54,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:48:54,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dd32.c.gcp-upstream.internal';"
2026-09-24 11:48:55,184 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 11:48:55,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 11:48:55,434 [I] DB_PATCHES have already been applied
I20260924 11:48:55.439863 1864 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-admin:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1DchkY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 11:48:55.439953 1864 mini_ranger.cc:193] Using host: 127.1.210.20
I20260924 11:48:55.454617 1864 mini_ranger.cc:227] Ranger admin URL: http://127.1.210.20:49685
Sep 24, 2026 11:48:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:48:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:48:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:48:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:48:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:48:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:48:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 11:48:56.611217 1864 mini_ranger.cc:160] Time spent starting Ranger: real 8.332s user 0.000s sys 0.035s
Sep 24, 2026 11:49:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:49:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 24, 2026 11:49:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:49:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:49:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 11:49:18.765393 1864 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/rangerkms_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/rangerkms_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.210.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 11:49:18 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250558, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 11:49:18.945333 1864 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 11:49:18.969259 1864 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 11:49:19.883483 1864 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 11:49:19.883662 1864 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms
I20260924 11:49:19.883796 1864 mini_ranger_kms.cc:227] Using postgres at 127.1.210.20:55461
2026-09-24 11:49:20,296 [I] DB FLAVOR :POSTGRES
2026-09-24 11:49:20,296 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:49:20,296 [I] Checking connection
2026-09-24 11:49:20,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 11:49:20,524 [I] connection success
2026-09-24 11:49:20,524 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 11:49:20,524 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 11:49:20,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 11:49:20,782 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 11:49:20,783 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:49:20,783 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 11:49:20,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 11:49:21,030 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 11:49:21.500934 1864 mini_ranger_kms.cc:310] Created kms service
I20260924 11:49:21.713461 1864 mini_ranger_kms.cc:326] Created kudu user
I20260924 11:49:21.757236 1864 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 11:49:21 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250558, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:49:22.170620 1864 mini_ranger_kms.cc:384] Added ranger policy
I20260924 11:49:22.170939 1864 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms:/tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 11:49:22.171028 1864 mini_ranger_kms.cc:240] Using host: 127.1.210.20
I20260924 11:49:22.189438 1864 mini_ranger_kms.cc:276] Ranger KMS PID: 13073
I20260924 11:49:22.189646 1864 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.210.20:34869
Sep 24, 2026 11:49:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:49:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:49:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:49:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:49:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:49:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 11:49:24.431098 1864 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.486s user 0.117s sys 0.313s
I20260924 11:49:24.431349 1864 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 11:49:24.431550 1864 mini_ranger_kms.cc:401] 127.1.210.20:34869/kms/v1/keys
Sep 24 11:49:24 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250564, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:49:25 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250564, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.20@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
Sep 24 11:49:25 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250564, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.20@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
Sep 24 11:49:27 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:49:27.764930 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36673
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:36673
--ranger_config_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:34869/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:49:28.089155 13124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:49:28.089419 13124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:49:28.089502 13124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:49:28.096832 13124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:49:28.096927 13124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:49:28.096989 13124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:49:28.097034 13124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:49:28.097072 13124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:49:28.106559 13124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.210.20:34869/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:36673
--ranger_config_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:49:28.107621 13124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:49:28.108770 13124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:49:28.120373 13132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:28.120460 13129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:28.120487 13130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:49:28.121361 13124 server_base.cc:1034] running on GCE node
I20260924 11:49:28.122138 13124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:49:28.123418 13124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:49:28.124650 13124 hybrid_clock.cc:648] HybridClock initialized: now 1790250568124622 us; error 44 us; skew 500 ppm
I20260924 11:49:28.125010 13124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:49:28 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:49:28.132918 13124 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:49:28.134310 13124 webserver.cc:460] Webserver started at http://127.1.210.62:40003/ using document root <none> and password file <none>
I20260924 11:49:28.134932 13124 fs_manager.cc:359] Metadata directory not provided
I20260924 11:49:28.135077 13124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:49:28.135365 13124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:49:28 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:49:28.200150 13124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b"
format_stamp: "Formatted at 2026-09-24 11:49:28 on dist-test-slave-dd32"
server_key: "27768a10e7edd2872a7432210708d9c5"
server_key_iv: "acc1bdbcdd0337e2983d5e23e6b0c965"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:28.200980 13124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b"
format_stamp: "Formatted at 2026-09-24 11:49:28 on dist-test-slave-dd32"
server_key: "27768a10e7edd2872a7432210708d9c5"
server_key_iv: "acc1bdbcdd0337e2983d5e23e6b0c965"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:28.206260 13124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:49:28.225273 13139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:28.226788 13124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:49:28.226977 13124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b"
format_stamp: "Formatted at 2026-09-24 11:49:28 on dist-test-slave-dd32"
server_key: "27768a10e7edd2872a7432210708d9c5"
server_key_iv: "acc1bdbcdd0337e2983d5e23e6b0c965"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:28.227676 13124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:49:28.246685 13124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:49:28.250763 13124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:49:28.251076 13124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:49:28.271970 13124 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:36673
I20260924 11:49:28.271979 13190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:36673 every 8 connection(s)
I20260924 11:49:28.273416 13124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:49:28.277347 13191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:49:28.278348 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 13124
I20260924 11:49:28.278544 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:49:28.279321 1864 external_mini_cluster.cc:1428] Setting key 0d5ca03acdc7f8ad005e180b2d22f3ef
I20260924 11:49:28.294478 13191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b: Bootstrap starting.
I20260924 11:49:28.298537 13191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b: Neither blocks nor log segments found. Creating new log.
I20260924 11:49:28.299620 13191 log.cc:824] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:49:28.303028 13191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b: No bootstrap required, opened a new log
Sep 24 11:49:28 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:49:28.312801 13191 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36673 } }
I20260924 11:49:28.313383 13191 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:49:28.313515 13191 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5ef1b0f4b941d2b9bb50beb3c7483b, State: Initialized, Role: FOLLOWER
I20260924 11:49:28.314980 13191 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36673 } }
I20260924 11:49:28.315268 13191 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:49:28.315402 13191 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:49:28.315588 13191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:49:28.318771 13191 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36673 } }
I20260924 11:49:28.319370 13191 leader_election.cc:302] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5ef1b0f4b941d2b9bb50beb3c7483b; no voters:
I20260924 11:49:28.319622 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:28.281428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.1.210.62:36673)
0924 11:49:28.296108 (+ 14680us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:28.296125 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:28.296177 (+ 52us) server_negotiation.cc:407] Connection header received
0924 11:49:28.298576 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:28.298604 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:28.299132 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:28.299570 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:28.301021 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:28.303006 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:28.304862 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:28.305933 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:28.306099 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:28.311119 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:28.311129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:28.311154 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:28.311195 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:28.315065 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:28.315879 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:28.315888 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:28.315901 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:28.316069 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:28.316505 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:28.316510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:28.316514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:28.317204 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:28.317495 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:28.317922 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 11:49:28.318286 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14433,"thread_start_us":15613,"threads_started":1}
I20260924 11:49:28.320165 13191 leader_election.cc:288] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:49:28.320294 13196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:49:28.325630 13191 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:49:28.327422 13196 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [term 1 LEADER]: Becoming Leader. State: Replica: cf5ef1b0f4b941d2b9bb50beb3c7483b, State: Running, Role: LEADER
I20260924 11:49:28.327975 13196 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36673 } }
I20260924 11:49:28.334478 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36673 } } }
I20260924 11:49:28.334827 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [sys.catalog]: This master's current role is: LEADER
I20260924 11:49:28.337294 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5ef1b0f4b941d2b9bb50beb3c7483b. Latest consensus state: current_term: 1 leader_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5ef1b0f4b941d2b9bb50beb3c7483b" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36673 } } }
I20260924 11:49:28.337569 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b [sys.catalog]: This master's current role is: LEADER
I20260924 11:49:28.345310 13203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:49:28.352011 13203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:49:28.371570 13203 catalog_manager.cc:1347] Generated new cluster ID: 9c5eff8f7c4442cf8c8d0ed2011f386c
I20260924 11:49:28.371798 13203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:49:28.384450 13203 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:49:28.386399 13203 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:49:28.391528 13191 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 11:49:28.394022 13203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b: Generated new TSK 0
I20260924 11:49:28.394752 13203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:49:29.257624 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:34869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36673
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:49:29 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 11:49:29.672194 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:49:29.672504 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:49:29.672688 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:49:29.680200 13236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:49:29.680348 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:49:29.680555 13236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:49:29.692188 13236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.210.20:34869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:49:29.693590 13236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:49:29.694895 13236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:49:29.714493 13246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:29.715682 13247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:29.723145 13249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:49:29.723657 13236 server_base.cc:1034] running on GCE node
I20260924 11:49:29.724447 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:49:29.725626 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:49:29.729120 13236 hybrid_clock.cc:648] HybridClock initialized: now 1790250569728988 us; error 88 us; skew 500 ppm
I20260924 11:49:29.729537 13236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:49:29 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:49:29.733541 13236 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:49:29.739099 13236 webserver.cc:460] Webserver started at http://127.1.210.1:40511/ using document root <none> and password file <none>
I20260924 11:49:29.739852 13236 fs_manager.cc:359] Metadata directory not provided
I20260924 11:49:29.740010 13236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:49:29.740345 13236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:49:29 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:49:29.762629 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "511e6791180d4769a7e24fd0e51df471"
format_stamp: "Formatted at 2026-09-24 11:49:29 on dist-test-slave-dd32"
server_key: "db823814b8fce15781f9db738e4dc669"
server_key_iv: "46d1949d431ad52273f695e6e8fbf270"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:29.763837 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "511e6791180d4769a7e24fd0e51df471"
format_stamp: "Formatted at 2026-09-24 11:49:29 on dist-test-slave-dd32"
server_key: "db823814b8fce15781f9db738e4dc669"
server_key_iv: "46d1949d431ad52273f695e6e8fbf270"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:29.782543 13236 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.008s sys 0.008s
I20260924 11:49:29.808755 13256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:29.815248 13236 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260924 11:49:29.815641 13236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "511e6791180d4769a7e24fd0e51df471"
format_stamp: "Formatted at 2026-09-24 11:49:29 on dist-test-slave-dd32"
server_key: "db823814b8fce15781f9db738e4dc669"
server_key_iv: "46d1949d431ad52273f695e6e8fbf270"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:29.816040 13236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:49:29.843652 13236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:49:29.848331 13236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:49:29.848626 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:49:29.849694 13236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:49:29.851171 13236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:49:29.851276 13236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:29.851429 13236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:49:29.851498 13236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:29.924101 13236 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:37689
I20260924 11:49:29.924118 13368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:37689 every 8 connection(s)
I20260924 11:49:29.925581 13236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:49:29.929526 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 13236
I20260924 11:49:29.929804 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:49:29.930824 1864 external_mini_cluster.cc:1428] Setting key f1a8123e92d6cb7dabd3f159a467ec43
Sep 24 11:49:29 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:49:30.009402 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:29.947208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36673 (local address 127.1.210.1:59693)
0924 11:49:29.968611 (+ 21403us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:29.968697 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:29.972724 (+ 4027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:29.983751 (+ 11027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:29.983765 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:29.984467 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:49:29.985407 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:29.985437 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:29.987029 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:29.987036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:29.989385 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:29.989402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:29.990812 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:29.990822 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:29.991187 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:29.992078 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:49:29.992111 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:49:29.995016 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:49:30.004443 (+ 9427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:49:30.004455 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:49:30.004476 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:49:30.005009 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:49:30.005480 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:49:30.005486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:49:30.005491 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:49:30.005680 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:49:30.006234 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:49:30.006244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:49:30.007014 (+ 770us) client_negotiation.cc:769] Sending connection context
0924 11:49:30.007358 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 11:49:30.007930 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20948,"thread_start_us":27520,"threads_started":1}
I20260924 11:49:30.010690 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:29.947960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:59693 (local address 127.1.210.62:36673)
0924 11:49:29.967516 (+ 19556us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:29.967530 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:29.972851 (+ 5321us) server_negotiation.cc:407] Connection header received
0924 11:49:29.983056 (+ 10205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:29.983062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:29.983176 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:29.983293 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:29.985622 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:29.986821 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:29.989566 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:29.990325 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:29.990435 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:29.995265 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:29.995270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:29.995273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:29.995313 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:30.004239 (+ 8926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:30.005211 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:30.005217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:30.005220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:30.005356 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:30.005809 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:30.005814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:30.005818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:30.006103 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:30.006282 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:30.010039 (+ 3757us) server_negotiation.cc:299] Negotiation successful
0924 11:49:30.010314 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19343,"thread_start_us":19581,"threads_started":1}
I20260924 11:49:30.012970 13369 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36673
I20260924 11:49:30.013382 13369 heartbeater.cc:464] Registering TS with master...
I20260924 11:49:30.014298 13369 heartbeater.cc:511] Master 127.1.210.62:36673 requested a full tablet report, sending...
I20260924 11:49:30.016892 13156 ts_manager.cc:194] Registered new tserver with Master: 511e6791180d4769a7e24fd0e51df471 (127.1.210.1:37689)
I20260924 11:49:30.018947 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:59693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:49:30.032614 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:34869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36673
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:49:30 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
Sep 24 11:49:30 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
W20260924 11:49:30.518392 13375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:49:30.518778 13375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:49:30.518937 13375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:49:30.530465 13375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:49:30.530591 13375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:49:30.530808 13375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:49:30.546445 13375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.210.20:34869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:49:30.548266 13375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:49:30.550047 13375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:49:30.594882 13382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:30.596251 13383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:30.603399 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:49:30.606745 13375 server_base.cc:1034] running on GCE node
I20260924 11:49:30.607527 13375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:49:30.608812 13375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:49:30.610117 13375 hybrid_clock.cc:648] HybridClock initialized: now 1790250570610029 us; error 75 us; skew 500 ppm
I20260924 11:49:30.610505 13375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:49:30 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:49:30.614938 13375 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:49:30.616925 13375 webserver.cc:460] Webserver started at http://127.1.210.2:34221/ using document root <none> and password file <none>
I20260924 11:49:30.617854 13375 fs_manager.cc:359] Metadata directory not provided
I20260924 11:49:30.617990 13375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:49:30.618352 13375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:49:30 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:49:30.637118 13375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a"
format_stamp: "Formatted at 2026-09-24 11:49:30 on dist-test-slave-dd32"
server_key: "1ac506a7564ad16e2c6ccc0bd5f8f040"
server_key_iv: "89bdc4cd679d1ededb8b896675914a45"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:30.638196 13375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a"
format_stamp: "Formatted at 2026-09-24 11:49:30 on dist-test-slave-dd32"
server_key: "1ac506a7564ad16e2c6ccc0bd5f8f040"
server_key_iv: "89bdc4cd679d1ededb8b896675914a45"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:30.645900 13375 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 11:49:30.666715 13397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:30.668535 13375 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260924 11:49:30.668704 13375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a"
format_stamp: "Formatted at 2026-09-24 11:49:30 on dist-test-slave-dd32"
server_key: "1ac506a7564ad16e2c6ccc0bd5f8f040"
server_key_iv: "89bdc4cd679d1ededb8b896675914a45"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:30.668948 13375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:49:30.698098 13375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:49:30.701702 13375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:49:30.701952 13375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:49:30.702797 13375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:49:30.704165 13375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:49:30.704262 13375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:30.704350 13375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:49:30.704393 13375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:30.739346 13375 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:46353
I20260924 11:49:30.739408 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:46353 every 8 connection(s)
I20260924 11:49:30.740660 13375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:49:30.744721 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 13375
I20260924 11:49:30.744894 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:49:30.745460 1864 external_mini_cluster.cc:1428] Setting key 30ef2c8d7c60fb440646e621ffd2da6a
Sep 24 11:49:30 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:49:30.765497 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:30.743200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:43629 (local address 127.1.210.62:36673)
0924 11:49:30.743632 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:30.743637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:30.745042 (+ 1405us) server_negotiation.cc:407] Connection header received
0924 11:49:30.747537 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:30.747543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:30.747633 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:30.747716 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:30.749708 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:30.750431 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:30.752926 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:30.753434 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:30.753529 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:30.757559 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:30.757568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:30.757576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:30.757631 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:30.762074 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:30.762853 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:30.762861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:30.762867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:30.763034 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:30.763463 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:30.763472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:30.763477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:30.763914 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:30.764056 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:30.764731 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 11:49:30.764969 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
I20260924 11:49:30.766057 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:30.744010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36673 (local address 127.1.210.2:43629)
0924 11:49:30.744875 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:30.744925 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:30.747110 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:30.748002 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:30.748012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:30.748599 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:49:30.749520 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:30.749548 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:30.750581 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:30.750589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:30.752766 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:30.752793 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:30.753646 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:30.753655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:30.753983 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:30.754689 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:49:30.754723 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:49:30.757355 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:49:30.762206 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:49:30.762212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:49:30.762224 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:49:30.762633 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:49:30.763144 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:49:30.763147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:49:30.763149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:49:30.763335 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:49:30.764037 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:49:30.764043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:49:30.764324 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 11:49:30.764543 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 11:49:30.764947 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260924 11:49:30.767491 13512 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36673
I20260924 11:49:30.767858 13512 heartbeater.cc:464] Registering TS with master...
I20260924 11:49:30.769284 13512 heartbeater.cc:511] Master 127.1.210.62:36673 requested a full tablet report, sending...
I20260924 11:49:30.770833 13156 ts_manager.cc:194] Registered new tserver with Master: ba1e76de8dc94685ae6d6cbb3d245e8a (127.1.210.2:46353)
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:49:30.771685 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:43629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:49:30.801630 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:34869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36673
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:49:31.022858 13369 heartbeater.cc:503] Master 127.1.210.62:36673 was elected leader, sending a full tablet report...
W20260924 11:49:31.131565 13518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:49:31.131906 13518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:49:31.132069 13518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:49:31.139071 13518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:49:31.139213 13518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:49:31.139416 13518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:49:31.149561 13518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.210.20:34869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:36673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:49:31.150888 13518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:49:31.152128 13518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:49:31.166070 13524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:31.166119 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:49:31.166146 13523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:49:31.167460 13518 server_base.cc:1034] running on GCE node
I20260924 11:49:31.168088 13518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:49:31.168951 13518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:49:31.170192 13518 hybrid_clock.cc:648] HybridClock initialized: now 1790250571170159 us; error 75 us; skew 500 ppm
I20260924 11:49:31.170506 13518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:49:31 dist-test-slave-dd32 krb5kdc[12680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250571, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:49:31.173521 13518 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:49:31.174891 13518 webserver.cc:460] Webserver started at http://127.1.210.3:46595/ using document root <none> and password file <none>
I20260924 11:49:31.175612 13518 fs_manager.cc:359] Metadata directory not provided
I20260924 11:49:31.175729 13518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:49:31.176060 13518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:49:31 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250571, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:49:31.192735 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "ae94088866c84f13af53061b7f991f1f"
format_stamp: "Formatted at 2026-09-24 11:49:31 on dist-test-slave-dd32"
server_key: "45849541975d7343075aa9a1c266c895"
server_key_iv: "0f7871b2753d284a31318ebc4b6d8883"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:31.193837 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "ae94088866c84f13af53061b7f991f1f"
format_stamp: "Formatted at 2026-09-24 11:49:31 on dist-test-slave-dd32"
server_key: "45849541975d7343075aa9a1c266c895"
server_key_iv: "0f7871b2753d284a31318ebc4b6d8883"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:31.200500 13518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 11:49:31.215268 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:31.216765 13518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:49:31.216940 13518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "ae94088866c84f13af53061b7f991f1f"
format_stamp: "Formatted at 2026-09-24 11:49:31 on dist-test-slave-dd32"
server_key: "45849541975d7343075aa9a1c266c895"
server_key_iv: "0f7871b2753d284a31318ebc4b6d8883"
server_key_version: "kuduclusterkey@0"
I20260924 11:49:31.217216 13518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:49:31.231356 13518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:49:31.234850 13518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:49:31.235136 13518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:49:31.236394 13518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:49:31.237701 13518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:49:31.237771 13518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:31.237931 13518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:49:31.237980 13518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:49:31.271847 13518 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:38677
I20260924 11:49:31.271950 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:38677 every 8 connection(s)
I20260924 11:49:31.273201 13518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:49:31.282552 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 13518
I20260924 11:49:31.282714 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:49:31.283140 1864 external_mini_cluster.cc:1428] Setting key 6faebf6bbd7759692d70838be84ce2bf
Sep 24 11:49:31 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250571, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:49:31.292796 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.275717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:36201 (local address 127.1.210.62:36673)
0924 11:49:31.276191 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.276197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.277240 (+ 1043us) server_negotiation.cc:407] Connection header received
0924 11:49:31.279381 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.279386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.279472 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.279570 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:31.280854 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.281638 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.283235 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.283711 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.283850 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.287487 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:31.287492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:31.287495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:31.287524 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:31.289979 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.290671 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.290683 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.290686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.290803 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.291226 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.291233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.291236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.291481 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:31.291636 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.292231 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.292431 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260924 11:49:31.293736 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.276332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:36673 (local address 127.1.210.3:36201)
0924 11:49:31.277087 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:31.277132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:31.279135 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:31.279767 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:31.279774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:31.280166 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:49:31.280717 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.280733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.281761 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.281764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.283090 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.283101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.283966 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.283973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.284237 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.285009 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:49:31.285030 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:49:31.287276 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:49:31.290119 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:49:31.290127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:49:31.290142 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:49:31.290543 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:49:31.290910 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:49:31.290917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:49:31.290924 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:49:31.291115 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:49:31.291621 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:49:31.291628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:49:31.291908 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 11:49:31.292159 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 11:49:31.292632 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
I20260924 11:49:31.294948 13646 heartbeater.cc:347] Connected to a master server at 127.1.210.62:36673
I20260924 11:49:31.295198 13646 heartbeater.cc:464] Registering TS with master...
I20260924 11:49:31.295794 13646 heartbeater.cc:511] Master 127.1.210.62:36673 requested a full tablet report, sending...
I20260924 11:49:31.297318 13156 ts_manager.cc:194] Registered new tserver with Master: ae94088866c84f13af53061b7f991f1f (127.1.210.3:38677)
I20260924 11:49:31.298465 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:49:31.298470 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:36201
I20260924 11:49:31.315418 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.303891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.1.210.62:36673)
0924 11:49:31.304050 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.304054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.304427 (+ 373us) server_negotiation.cc:407] Connection header received
0924 11:49:31.304597 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.304602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.304675 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.304765 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:31.306125 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.306794 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.308061 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.308637 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.308728 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.310865 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:31.310870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:31.310873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:31.310900 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:31.313435 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.313981 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.313984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.313986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.314087 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.314375 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.314378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.314380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.314621 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:31.314781 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.315057 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.315200 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:49:31.319046 13156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:49:31.327004 13156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:49:31.359758 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.340021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.1.210.3:38677)
0924 11:49:31.350414 (+ 10393us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.350428 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.351644 (+ 1216us) server_negotiation.cc:407] Connection header received
0924 11:49:31.351753 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.351764 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.352052 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.352231 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:49:31.353330 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.354718 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.357281 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.358259 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.358413 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.358658 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.359243 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.359408 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10189,"thread_start_us":10417,"threads_started":1}
I20260924 11:49:31.359771 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.339361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38677 (local address 127.0.0.1:41626)
0924 11:49:31.351187 (+ 11826us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:31.351238 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:31.351436 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:31.352561 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:31.352572 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:31.352613 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:49:31.353089 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.353105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.354857 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.354861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.357123 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.357153 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.358805 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.358812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.358988 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.359036 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 11:49:31.359156 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 11:49:31.359331 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11553,"thread_start_us":11350,"threads_started":1}
I20260924 11:49:31.362288 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.339368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:37689 (local address 127.0.0.1:52928)
0924 11:49:31.352620 (+ 13252us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:31.352654 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:31.352753 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:31.354233 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:31.354238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:31.354262 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:49:31.354604 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.354614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.356579 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.356585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.359586 (+ 3001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.359599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.361436 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.361443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.361617 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.361655 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:49:31.361756 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:49:31.361881 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13133,"mutex_wait_us":389,"thread_start_us":12943,"threads_started":1}
I20260924 11:49:31.362950 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.339475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.1.210.1:37689)
0924 11:49:31.353693 (+ 14218us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.353701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.353727 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:49:31.353820 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.353826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.354035 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.354220 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:49:31.354837 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.356446 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.359791 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.361231 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.361372 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.361593 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.362404 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.362571 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14037,"thread_start_us":13603,"threads_started":1}
I20260924 11:49:31.363399 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.341680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:46353 (local address 127.0.0.1:52194)
0924 11:49:31.353506 (+ 11826us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:31.353524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:31.353633 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:31.354508 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:31.354513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:31.354540 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:49:31.355039 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.355055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.356451 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.356459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.360221 (+ 3762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.360235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.361753 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.361758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.361932 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.361967 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:49:31.362059 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 11:49:31.362178 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11712,"mutex_wait_us":81,"thread_start_us":11562,"threads_started":1}
I20260924 11:49:31.363405 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.342219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.1.210.2:46353)
0924 11:49:31.353228 (+ 11009us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.353238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.353747 (+ 509us) server_negotiation.cc:407] Connection header received
0924 11:49:31.353832 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.353838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.354033 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.354220 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:49:31.355260 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.356304 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.360347 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.361418 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.361543 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.361712 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.362133 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.362861 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10785,"thread_start_us":10697,"threads_started":1}
I20260924 11:49:31.363997 13581 tablet_service.cc:1467] Processing CreateTablet for tablet dae41310b72e4c90a131185f66365267 (DEFAULT_TABLE table=test-table [id=28c6f6a4e81e465f98f687b6c3733672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:49:31.365573 13581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae41310b72e4c90a131185f66365267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:49:31.366727 13447 tablet_service.cc:1467] Processing CreateTablet for tablet dae41310b72e4c90a131185f66365267 (DEFAULT_TABLE table=test-table [id=28c6f6a4e81e465f98f687b6c3733672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:49:31.366854 13286 tablet_service.cc:1467] Processing CreateTablet for tablet dae41310b72e4c90a131185f66365267 (DEFAULT_TABLE table=test-table [id=28c6f6a4e81e465f98f687b6c3733672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:49:31.368031 13447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae41310b72e4c90a131185f66365267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:49:31.368214 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae41310b72e4c90a131185f66365267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:49:31.379964 13665 tablet_bootstrap.cc:492] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Bootstrap starting.
I20260924 11:49:31.381897 13666 tablet_bootstrap.cc:492] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Bootstrap starting.
I20260924 11:49:31.384872 13665 tablet_bootstrap.cc:654] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Neither blocks nor log segments found. Creating new log.
I20260924 11:49:31.385392 13666 tablet_bootstrap.cc:654] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Neither blocks nor log segments found. Creating new log.
I20260924 11:49:31.385583 13667 tablet_bootstrap.cc:492] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Bootstrap starting.
I20260924 11:49:31.386770 13665 log.cc:824] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Log is configured to *not* fsync() on all Append() calls
I20260924 11:49:31.387127 13666 log.cc:824] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:49:31.388785 13667 tablet_bootstrap.cc:654] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Neither blocks nor log segments found. Creating new log.
I20260924 11:49:31.389935 13665 tablet_bootstrap.cc:492] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: No bootstrap required, opened a new log
I20260924 11:49:31.389991 13667 log.cc:824] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:49:31.390216 13666 tablet_bootstrap.cc:492] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: No bootstrap required, opened a new log
I20260924 11:49:31.390218 13665 ts_tablet_manager.cc:1397] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:49:31.390494 13666 ts_tablet_manager.cc:1397] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:49:31.393397 13667 tablet_bootstrap.cc:492] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: No bootstrap required, opened a new log
I20260924 11:49:31.393648 13667 ts_tablet_manager.cc:1397] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260924 11:49:31.397957 13665 raft_consensus.cc:348] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.398327 13665 raft_consensus.cc:374] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:49:31.398429 13665 raft_consensus.cc:729] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511e6791180d4769a7e24fd0e51df471, State: Initialized, Role: FOLLOWER
I20260924 11:49:31.399039 13665 consensus_queue.cc:260] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.401258 13666 raft_consensus.cc:348] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.401726 13666 raft_consensus.cc:374] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:49:31.401823 13666 raft_consensus.cc:729] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae94088866c84f13af53061b7f991f1f, State: Initialized, Role: FOLLOWER
I20260924 11:49:31.402472 13665 ts_tablet_manager.cc:1428] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 11:49:31.402448 13666 consensus_queue.cc:260] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.403714 13646 heartbeater.cc:503] Master 127.1.210.62:36673 was elected leader, sending a full tablet report...
I20260924 11:49:31.404286 13666 ts_tablet_manager.cc:1428] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 11:49:31.405083 13667 raft_consensus.cc:348] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.405483 13667 raft_consensus.cc:374] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:49:31.405596 13667 raft_consensus.cc:729] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1e76de8dc94685ae6d6cbb3d245e8a, State: Initialized, Role: FOLLOWER
I20260924 11:49:31.406203 13667 consensus_queue.cc:260] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.407243 13512 heartbeater.cc:503] Master 127.1.210.62:36673 was elected leader, sending a full tablet report...
I20260924 11:49:31.408988 13667 ts_tablet_manager.cc:1428] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260924 11:49:31.484124 13370 tablet.cc:2369] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:49:31.492792 13513 tablet.cc:2369] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:49:31.524991 13647 tablet.cc:2369] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:49:31.599834 13672 raft_consensus.cc:482] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:49:31.600159 13672 raft_consensus.cc:504] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.601974 13672 leader_election.cc:288] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 511e6791180d4769a7e24fd0e51df471 (127.1.210.1:37689), ba1e76de8dc94685ae6d6cbb3d245e8a (127.1.210.2:46353)
I20260924 11:49:31.608649 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.602176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:37689 (local address 127.1.210.3:47211)
0924 11:49:31.602485 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:31.602509 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:31.602631 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:31.603078 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:31.603085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:31.603123 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:49:31.603322 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.603333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.604620 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.604626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.606158 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.606169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.607872 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.607877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.608078 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.608109 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:49:31.608208 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 11:49:31.608318 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 11:49:31.608675 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.602386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:47211 (local address 127.1.210.1:37689)
0924 11:49:31.602581 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.602587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.602607 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:49:31.602815 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.602822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.602909 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.603013 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:49:31.603488 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.604488 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.606311 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.607723 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.607820 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.607893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.608274 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.608419 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:49:31.608753 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.602481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:39391 (local address 127.1.210.2:46353)
0924 11:49:31.602723 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.602732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.603309 (+ 577us) server_negotiation.cc:407] Connection header received
0924 11:49:31.603428 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.603433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.603508 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.603573 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:49:31.604157 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.605023 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.606986 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.607874 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.608007 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.608102 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.608374 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.608523 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:49:31.608992 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.602405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:46353 (local address 127.1.210.3:39391)
0924 11:49:31.603198 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 11:49:31.603211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:49:31.603309 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:49:31.603625 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:49:31.603630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:49:31.603655 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:49:31.604011 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.604021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.605194 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.605197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.606820 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:49:31.606833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.608061 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.608065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:49:31.608181 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.608238 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 11:49:31.608328 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 11:49:31.608443 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":439,"threads_started":1}
I20260924 11:49:31.609760 13324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae41310b72e4c90a131185f66365267" candidate_uuid: "ae94088866c84f13af53061b7f991f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511e6791180d4769a7e24fd0e51df471" is_pre_election: true
I20260924 11:49:31.609912 13467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae41310b72e4c90a131185f66365267" candidate_uuid: "ae94088866c84f13af53061b7f991f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" is_pre_election: true
I20260924 11:49:31.610276 13324 raft_consensus.cc:2454] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae94088866c84f13af53061b7f991f1f in term 0.
I20260924 11:49:31.610387 13467 raft_consensus.cc:2454] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae94088866c84f13af53061b7f991f1f in term 0.
I20260924 11:49:31.610878 13534 leader_election.cc:302] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 511e6791180d4769a7e24fd0e51df471, ae94088866c84f13af53061b7f991f1f; no voters:
I20260924 11:49:31.611210 13672 raft_consensus.cc:2781] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:49:31.611349 13672 raft_consensus.cc:482] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:49:31.611459 13672 raft_consensus.cc:3037] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:49:31.613636 13672 raft_consensus.cc:504] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.614207 13672 leader_election.cc:288] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [CANDIDATE]: Term 1 election: Requested vote from peers 511e6791180d4769a7e24fd0e51df471 (127.1.210.1:37689), ba1e76de8dc94685ae6d6cbb3d245e8a (127.1.210.2:46353)
I20260924 11:49:31.614532 13324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae41310b72e4c90a131185f66365267" candidate_uuid: "ae94088866c84f13af53061b7f991f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511e6791180d4769a7e24fd0e51df471"
I20260924 11:49:31.614668 13467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae41310b72e4c90a131185f66365267" candidate_uuid: "ae94088866c84f13af53061b7f991f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a"
I20260924 11:49:31.614706 13324 raft_consensus.cc:3037] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:49:31.614861 13467 raft_consensus.cc:3037] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:49:31.616784 13324 raft_consensus.cc:2454] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae94088866c84f13af53061b7f991f1f in term 1.
I20260924 11:49:31.616984 13467 raft_consensus.cc:2454] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae94088866c84f13af53061b7f991f1f in term 1.
I20260924 11:49:31.617168 13534 leader_election.cc:302] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 511e6791180d4769a7e24fd0e51df471, ae94088866c84f13af53061b7f991f1f; no voters:
I20260924 11:49:31.617477 13672 raft_consensus.cc:2781] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:49:31.617870 13672 raft_consensus.cc:686] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 1 LEADER]: Becoming Leader. State: Replica: ae94088866c84f13af53061b7f991f1f, State: Running, Role: LEADER
I20260924 11:49:31.618412 13672 consensus_queue.cc:237] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } }
I20260924 11:49:31.622956 13156 catalog_manager.cc:5495] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to ae94088866c84f13af53061b7f991f1f (127.1.210.3). New cstate: current_term: 1 leader_uuid: "ae94088866c84f13af53061b7f991f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae94088866c84f13af53061b7f991f1f" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 } health_report { overall_health: UNKNOWN } } }
I20260924 11:49:31.654382 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.649568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.1.210.3:38677)
0924 11:49:31.649775 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.649779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.649797 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:49:31.649969 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.649972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.650054 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.650174 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:49:31.650506 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.651185 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.652520 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.652891 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.652978 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.653384 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:49:31.653865 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:49:31.653969 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.654036 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.654092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:49:31.662112 13467 raft_consensus.cc:1261] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 1 FOLLOWER]: Refusing update from remote peer ae94088866c84f13af53061b7f991f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:49:31.662129 13324 raft_consensus.cc:1261] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 1 FOLLOWER]: Refusing update from remote peer ae94088866c84f13af53061b7f991f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:49:31.663060 13675 consensus_queue.cc:1035] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba1e76de8dc94685ae6d6cbb3d245e8a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:49:31.663407 13672 consensus_queue.cc:1035] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "511e6791180d4769a7e24fd0e51df471" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:49:31.674775 13679 mvcc.cc:205] Tried to move back new op lower bound from 7332866341518934016 to 7332866341352677376. Current Snapshot: MvccSnapshot[applied={T|T < 7332866341518934016}]
I20260924 11:49:31.675673 13678 mvcc.cc:205] Tried to move back new op lower bound from 7332866341518934016 to 7332866341352677376. Current Snapshot: MvccSnapshot[applied={T|T < 7332866341518934016}]
I20260924 11:49:31.687932 13156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:49:31.688125 13156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:49:31.691525 13156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf5ef1b0f4b941d2b9bb50beb3c7483b: Sending DeleteTablet for 3 replicas of tablet dae41310b72e4c90a131185f66365267
I20260924 11:49:31.692585 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet dae41310b72e4c90a131185f66365267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:49:31 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:52928
I20260924 11:49:31.692587 13581 tablet_service.cc:1514] Processing DeleteTablet for tablet dae41310b72e4c90a131185f66365267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:49:31 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:41626
I20260924 11:49:31.692776 13447 tablet_service.cc:1514] Processing DeleteTablet for tablet dae41310b72e4c90a131185f66365267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:49:31 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:52194
I20260924 11:49:31.695475 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 13236
I20260924 11:49:31.704846 13688 tablet_replica.cc:330] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: stopping tablet replica
I20260924 11:49:31.705430 13689 tablet_replica.cc:330] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: stopping tablet replica
I20260924 11:49:31.706390 13689 raft_consensus.cc:2229] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:49:31.705957 13688 raft_consensus.cc:2229] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:49:31.707121 13689 raft_consensus.cc:2258] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:49:31.707634 13688 raft_consensus.cc:2258] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:49:31.708832 13690 tablet_replica.cc:330] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: stopping tablet replica
I20260924 11:49:31.709300 13690 raft_consensus.cc:2229] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:49:31.709793 13690 raft_consensus.cc:2258] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:49:31.710287 13689 ts_tablet_manager.cc:1905] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:49:31.711894 13690 ts_tablet_manager.cc:1905] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:49:31.719015 13690 ts_tablet_manager.cc:1918] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:49:31.719247 13690 log.cc:1196] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/dae41310b72e4c90a131185f66365267
I20260924 11:49:31.719229 13688 ts_tablet_manager.cc:1905] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:49:31.719882 13690 ts_tablet_manager.cc:1939] T dae41310b72e4c90a131185f66365267 P ba1e76de8dc94685ae6d6cbb3d245e8a: Deleting consensus metadata
I20260924 11:49:31.721849 13141 catalog_manager.cc:4841] TS ba1e76de8dc94685ae6d6cbb3d245e8a (127.1.210.2:46353): tablet dae41310b72e4c90a131185f66365267 (table test-table [id=28c6f6a4e81e465f98f687b6c3733672]) successfully deleted
I20260924 11:49:31.726437 13689 ts_tablet_manager.cc:1918] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:49:31.726684 13689 log.cc:1196] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/dae41310b72e4c90a131185f66365267
I20260924 11:49:31.727198 13688 ts_tablet_manager.cc:1918] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:49:31.727355 13688 log.cc:1196] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/dae41310b72e4c90a131185f66365267
I20260924 11:49:31.727479 13689 ts_tablet_manager.cc:1939] T dae41310b72e4c90a131185f66365267 P ae94088866c84f13af53061b7f991f1f: Deleting consensus metadata
I20260924 11:49:31.727826 13688 ts_tablet_manager.cc:1939] T dae41310b72e4c90a131185f66365267 P 511e6791180d4769a7e24fd0e51df471: Deleting consensus metadata
I20260924 11:49:31.729403 13142 catalog_manager.cc:4841] TS ae94088866c84f13af53061b7f991f1f (127.1.210.3:38677): tablet dae41310b72e4c90a131185f66365267 (table test-table [id=28c6f6a4e81e465f98f687b6c3733672]) successfully deleted
I20260924 11:49:31.729653 13140 catalog_manager.cc:4841] TS 511e6791180d4769a7e24fd0e51df471 (127.1.210.1:37689): tablet dae41310b72e4c90a131185f66365267 (table test-table [id=28c6f6a4e81e465f98f687b6c3733672]) successfully deleted
Sep 24 11:49:31 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:49:31.741452 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.697941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.1.210.1:37689)
0924 11:49:31.698164 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.698173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.722981 (+ 24808us) server_negotiation.cc:407] Connection header received
0924 11:49:31.723590 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.723601 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.723681 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.723818 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:31.725580 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.726744 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.728825 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.729517 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.729657 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.734762 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:31.734774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:31.734788 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:31.734825 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:31.737842 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.739150 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.739162 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.739165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.739318 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.739782 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.739789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.739791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.740231 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:31.740528 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.740834 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.741048 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:49:31.742127 13364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:49:31.865238 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 13236
I20260924 11:49:31.889345 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 13375
Sep 24 11:49:31 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:49:31.906523 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:31.890294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.1.210.2:46353)
0924 11:49:31.890468 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:31.890472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:31.890495 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:49:31.890690 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:31.890694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:31.890775 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:31.890868 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:31.892481 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.893376 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.895201 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:31.895624 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:31.895745 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:31.900754 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:31.900767 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:31.900782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:31.900824 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:31.903748 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.904442 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.904449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.904451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.904575 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:31.904952 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:31.904958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:31.904961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:31.905364 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:31.905662 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:31.905967 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 11:49:31.906181 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:49:31.907121 13507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:49:31.996399 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 13375
I20260924 11:49:32.018895 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 13518
Sep 24 11:49:32 dist-test-slave-dd32 krb5kdc[12680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:49:32.036535 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:49:32.019608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.1.210.3:38677)
0924 11:49:32.019770 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:49:32.019774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:49:32.019842 (+ 68us) server_negotiation.cc:407] Connection header received
0924 11:49:32.020035 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:49:32.020038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:49:32.020138 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:49:32.020219 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:49:32.022025 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:32.022835 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:32.024685 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:49:32.025195 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:49:32.025292 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:49:32.030471 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:49:32.030482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:49:32.030500 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:49:32.030540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:49:32.033732 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:32.034414 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:32.034422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:32.034424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:32.034570 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:49:32.034943 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:49:32.034948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:49:32.034951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:49:32.035289 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:49:32.035614 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 11:49:32.035922 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 11:49:32.036134 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:49:32.037213 13641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:49:32.157347 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 13518
I20260924 11:49:32.176381 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 13124
I20260924 11:49:32.177644 13186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:49:32.288401 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 13124
I20260924 11:49:32.308472 1864 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 11:49:32.645160 1864 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 11:49:32.645354 1864 mini_ranger.cc:66] Stopping Ranger...
I20260924 11:49:32.992112 1864 mini_ranger.cc:68] Stopped Ranger
2026-09-24 11:49:32.992 UTC [12710] LOG: received smart shutdown request
2026-09-24 11:49:32.995 UTC [12710] LOG: background worker "logical replication launcher" (PID 12719) exited with exit code 1
2026-09-24 11:49:32.996 UTC [12714] LOG: shutting down
2026-09-24 11:49:33.156 UTC [12710] LOG: database system is shut down
2026-09-24T11:49:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49848 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:49:33 dist-test-slave-dd32 krb5kdc[13704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:49:33 dist-test-slave-dd32 krb5kdc[13704](info): set up 2 sockets
Sep 24 11:49:33 dist-test-slave-dd32 krb5kdc[13704](info): commencing operation
krb5kdc: starting...
W20260924 11:49:34.385398 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:49:34 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:49:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/rangeradmin_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/rangeradmin_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/rangerlookup_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/rangerlookup_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
I20260924 11:49:34.697801 1864 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1DchkY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/postgres -l logfile start
2026-09-24 11:49:37.123 UTC [13734] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 11:49:37.123 UTC [13734] LOG: listening on IPv4 address "127.1.210.20", port 55323
2026-09-24 11:49:37.127 UTC [13734] LOG: listening on Unix socket "/tmp/.s.PGSQL.55323"
2026-09-24 11:49:37.149 UTC [13737] LOG: database system was shut down at 2026-09-24 11:49:35 UTC
2026-09-24 11:49:37.152 UTC [13734] LOG: database system is ready to accept connections
I20260924 11:49:38.117125 1864 mini_postgres.cc:96] Postgres bound to 55323
2026-09-24 11:49:38.137 UTC [13745] FATAL: database "slave" does not exist
127.1.210.20:55323 - accepting connections
I20260924 11:49:38.138427 1864 mini_ranger.cc:161] Starting Ranger...
I20260924 11:49:38.163075 1864 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 11:49:38.633677 1864 mini_ranger.cc:87] Created ranger Postgres database
I20260924 11:49:38.633791 1864 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-admin
2026-09-24 11:49:38,944 [I] DB FLAVOR :POSTGRES
2026-09-24 11:49:38,945 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:49:38,945 [I] Checking connection..
2026-09-24 11:49:38,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 11:49:39,202 [I] Checking connection passed.
2026-09-24 11:49:39,203 [I] --------- Verifying version history table ---------
2026-09-24 11:49:39,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:49:39,463 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:49:39,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:49:39,724 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:49:39,724 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 11:49:39,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 11:49:39.945 UTC [13809] WARNING: there is no transaction in progress
2026-09-24 11:49:39,951 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 11:49:39,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:49:40,184 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 11:49:40,184 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:49:40,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 11:49:40,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 11:49:40,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dd32.c.gcp-upstream.internal','N') ;"
2026-09-24 11:49:40,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:49:41,115 [I] Table x_portal_user does not exist in database ranger
2026-09-24 11:49:41,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:49:41,374 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 11:49:41,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:49:41,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:49:41,881 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 11:49:41,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 11:49:42.778 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.794 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.809 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.824 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.839 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.858 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.910 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.916 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.932 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.942 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.952 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.965 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.976 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.982 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.989 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:42.996 UTC [13917] WARNING: there is no transaction in progress
2026-09-24 11:49:43,599 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 11:49:43,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 11:49:43,849 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 11:49:43,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:49:44,108 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 11:49:44,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:49:44,364 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 11:49:44,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:49:44,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:49:44,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dd32.c.gcp-upstream.internal';"
2026-09-24 11:49:45,094 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 11:49:45,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 11:49:45,330 [I] DB_PATCHES have already been applied
I20260924 11:49:45.335889 1864 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-admin:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1DchkY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 11:49:45.336004 1864 mini_ranger.cc:193] Using host: 127.1.210.20
I20260924 11:49:45.349968 1864 mini_ranger.cc:227] Ranger admin URL: http://127.1.210.20:45741
Sep 24, 2026 11:49:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:49:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:49:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:49:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:49:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:49:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:49:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 11:49:46.495883 1864 mini_ranger.cc:160] Time spent starting Ranger: real 8.357s user 0.000s sys 0.036s
Sep 24, 2026 11:50:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:50:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 11:50:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:50:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:50:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 11:50:07.847812 1864 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.210.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.210.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/rangerkms_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.210.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/rangerkms_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.210.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/HTTP_127.1.210.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 11:50:07 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250607, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 11:50:08.003674 1864 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 11:50:08.025516 1864 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 11:50:08.892936 1864 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 11:50:08.893052 1864 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms
I20260924 11:50:08.893127 1864 mini_ranger_kms.cc:227] Using postgres at 127.1.210.20:55323
2026-09-24 11:50:09,254 [I] DB FLAVOR :POSTGRES
2026-09-24 11:50:09,255 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:50:09,255 [I] Checking connection
2026-09-24 11:50:09,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 11:50:09,480 [I] connection success
2026-09-24 11:50:09,481 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 11:50:09,481 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 11:50:09,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 11:50:09,718 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 11:50:09,719 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:50:09,719 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 11:50:09,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.20:55323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 11:50:09,977 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 11:50:10.429602 1864 mini_ranger_kms.cc:310] Created kms service
I20260924 11:50:10.603058 1864 mini_ranger_kms.cc:326] Created kudu user
I20260924 11:50:10.646951 1864 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 11:50:10 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250607, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:11.043431 1864 mini_ranger_kms.cc:384] Added ranger policy
I20260924 11:50:11.043772 1864 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms:/tmp/dist-test-task1DchkY/build/asan/bin/postgresql.jar:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1DchkY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 11:50:11.043845 1864 mini_ranger_kms.cc:240] Using host: 127.1.210.20
I20260924 11:50:11.063575 1864 mini_ranger_kms.cc:276] Ranger KMS PID: 14097
I20260924 11:50:11.063710 1864 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.210.20:45345
Sep 24, 2026 11:50:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 11:50:13 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250613, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:13.345779 1864 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.342s user 0.056s sys 0.322s
I20260924 11:50:13.346021 1864 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 11:50:13.346189 1864 mini_ranger_kms.cc:401] 127.1.210.20:45345/kms/v1/keys
Sep 24 11:50:13 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250613, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.20@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
Sep 24 11:50:13 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250613, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.20@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
Sep 24 11:50:16 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:50:16.160611 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37511
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:37511
--ranger_config_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:16.483420 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:16.483693 14148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:16.483843 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:16.491173 14148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:16.491283 14148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:16.491366 14148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:16.491472 14148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:16.491559 14148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:16.501832 14148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.210.20:45345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:37511
--ranger_config_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:16.503095 14148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:16.504465 14148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:16.518392 14153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:16.518668 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:16.518877 14154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:16.519179 14148 server_base.cc:1034] running on GCE node
I20260924 11:50:16.520061 14148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:16.521631 14148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:16.522884 14148 hybrid_clock.cc:648] HybridClock initialized: now 1790250616522833 us; error 42 us; skew 500 ppm
I20260924 11:50:16.523258 14148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:16 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:16.531327 14148 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:16.533116 14148 webserver.cc:460] Webserver started at http://127.1.210.62:35643/ using document root <none> and password file <none>
I20260924 11:50:16.533828 14148 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:16.533982 14148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:16.534327 14148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:50:16 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:16.595196 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "cdeb351abefd42c09e465c7d5c02d718"
format_stamp: "Formatted at 2026-09-24 11:50:16 on dist-test-slave-dd32"
server_key: "fd38b91e10e4e2beb46f886d56002657"
server_key_iv: "75acd0c26660a5e157fd2f14d29d6ff6"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:16.596274 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "cdeb351abefd42c09e465c7d5c02d718"
format_stamp: "Formatted at 2026-09-24 11:50:16 on dist-test-slave-dd32"
server_key: "fd38b91e10e4e2beb46f886d56002657"
server_key_iv: "75acd0c26660a5e157fd2f14d29d6ff6"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:16.603070 14148 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 11:50:16.622855 14163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:16.624644 14148 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:50:16.624832 14148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "cdeb351abefd42c09e465c7d5c02d718"
format_stamp: "Formatted at 2026-09-24 11:50:16 on dist-test-slave-dd32"
server_key: "fd38b91e10e4e2beb46f886d56002657"
server_key_iv: "75acd0c26660a5e157fd2f14d29d6ff6"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:16.625099 14148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:16.673650 14148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:16.678607 14148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:16.678965 14148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:16.699546 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:37511 every 8 connection(s)
I20260924 11:50:16.699568 14148 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:37511
I20260924 11:50:16.701424 14148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:16.703171 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14148
I20260924 11:50:16.703405 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:50:16.704236 1864 external_mini_cluster.cc:1428] Setting key d71293343acec8949e45a2477c2a0c7d
I20260924 11:50:16.706404 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:16.728190 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Bootstrap starting.
I20260924 11:50:16.731871 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:16.733253 14215 log.cc:824] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:50:16 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:16.736461 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: No bootstrap required, opened a new log
I20260924 11:50:16.746573 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:16.706385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.1.210.62:37511)
0924 11:50:16.721654 (+ 15269us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:16.721672 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:16.721727 (+ 55us) server_negotiation.cc:407] Connection header received
0924 11:50:16.724221 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:16.724256 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:16.724866 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:16.725389 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:16.726933 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:16.728976 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:16.731020 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:16.732065 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:16.732248 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:16.737267 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:16.737277 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:16.737296 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:16.737337 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:16.741469 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:16.742347 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:16.742361 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:16.742370 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:16.742559 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:16.743017 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:16.743027 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:16.743031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:16.743822 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:16.744115 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:16.744562 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 11:50:16.745017 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15002,"thread_start_us":14781,"threads_started":1}
I20260924 11:50:16.747340 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:16.747792 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:16.748023 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdeb351abefd42c09e465c7d5c02d718, State: Initialized, Role: FOLLOWER
I20260924 11:50:16.748739 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:16.749004 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:16.749174 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:16.749344 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:16.751652 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:16.752151 14215 leader_election.cc:302] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdeb351abefd42c09e465c7d5c02d718; no voters:
I20260924 11:50:16.752802 14215 leader_election.cc:288] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:50:16.752897 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:16.755208 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:16.759809 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 LEADER]: Becoming Leader. State: Replica: cdeb351abefd42c09e465c7d5c02d718, State: Running, Role: LEADER
I20260924 11:50:16.760648 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:16.764546 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdeb351abefd42c09e465c7d5c02d718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } } }
I20260924 11:50:16.764991 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:16.766188 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdeb351abefd42c09e465c7d5c02d718. Latest consensus state: current_term: 1 leader_uuid: "cdeb351abefd42c09e465c7d5c02d718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } } }
I20260924 11:50:16.766556 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:16.779248 14228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:16.787225 14228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:16.788040 14215 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 11:50:16.816885 14228 catalog_manager.cc:1347] Generated new cluster ID: 1ed4de45a96f46bcab223ae3234d7719
I20260924 11:50:16.817265 14228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:16.854831 14228 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:50:16.856931 14228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:16.871484 14228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Generated new TSK 0
I20260924 11:50:16.872666 14228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:50:17.710742 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37511
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:50:17 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250617, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 11:50:18.159153 14261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:18.159448 14261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:18.159615 14261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:18.167426 14261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:18.167541 14261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:18.167722 14261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:50:18.178035 14261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.210.20:45345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:18.179349 14261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:18.180701 14261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:18.205158 14271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:18.206036 14272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:18.215543 14261 server_base.cc:1034] running on GCE node
W20260924 11:50:18.216281 14274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:18.217203 14261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:18.219151 14261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:18.222579 14261 hybrid_clock.cc:648] HybridClock initialized: now 1790250618222457 us; error 99 us; skew 500 ppm
I20260924 11:50:18.222992 14261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:18 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:18.227448 14261 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:50:18.230815 14261 webserver.cc:460] Webserver started at http://127.1.210.1:33277/ using document root <none> and password file <none>
I20260924 11:50:18.231767 14261 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:18.231943 14261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:18.232339 14261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:50:18 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:18.253412 14261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "80958090704b467b8affe0c4b0cb45ad"
format_stamp: "Formatted at 2026-09-24 11:50:18 on dist-test-slave-dd32"
server_key: "af3a0196778c3745f249387fed73ef46"
server_key_iv: "f3f52526ca6b79e11bbb82d07872302c"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:18.255679 14261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "80958090704b467b8affe0c4b0cb45ad"
format_stamp: "Formatted at 2026-09-24 11:50:18 on dist-test-slave-dd32"
server_key: "af3a0196778c3745f249387fed73ef46"
server_key_iv: "f3f52526ca6b79e11bbb82d07872302c"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:18.263022 14261 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260924 11:50:18.300834 14281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:18.303009 14261 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.011s sys 0.009s
I20260924 11:50:18.303236 14261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "80958090704b467b8affe0c4b0cb45ad"
format_stamp: "Formatted at 2026-09-24 11:50:18 on dist-test-slave-dd32"
server_key: "af3a0196778c3745f249387fed73ef46"
server_key_iv: "f3f52526ca6b79e11bbb82d07872302c"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:18.303499 14261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:18.327649 14261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:18.332803 14261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:18.333273 14261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:18.335417 14261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:18.336980 14261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:18.337098 14261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:18.337244 14261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:18.337328 14261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:50:18 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250617, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:18.477873 14261 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:36347
I20260924 11:50:18.477924 14393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:36347 every 8 connection(s)
I20260924 11:50:18.479420 14261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:50:18.481588 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14261
I20260924 11:50:18.481769 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:50:18.482569 1864 external_mini_cluster.cc:1428] Setting key 85102bbc5da61d6fd8631255c759c56c
Sep 24 11:50:18 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
I20260924 11:50:18.547294 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:18.483732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:53929 (local address 127.1.210.62:37511)
0924 11:50:18.520769 (+ 37037us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:18.520775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:18.520802 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:50:18.524886 (+ 4084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:18.524893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:18.525000 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:18.525135 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:18.530374 (+ 5239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:18.531591 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:18.534014 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:18.534668 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:18.534775 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:18.540324 (+ 5549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:18.540330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:18.540335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:18.540372 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:18.543716 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:18.544596 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:18.544602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:18.544606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:18.544764 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:18.545310 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:18.545315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:18.545319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:18.545650 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:18.545802 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:18.546686 (+ 884us) server_negotiation.cc:299] Negotiation successful
0924 11:50:18.546902 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36897,"thread_start_us":39616,"threads_started":1}
I20260924 11:50:18.549592 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:18.482793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37511 (local address 127.1.210.1:53929)
0924 11:50:18.519469 (+ 36676us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:18.519542 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:18.524505 (+ 4963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:18.525781 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:18.525795 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:18.526541 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:18.527406 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:18.527433 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:18.531806 (+ 4373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:18.531811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:18.533847 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:18.533862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:18.535131 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:18.535139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:18.535441 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:18.536315 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:18.536345 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:18.539177 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:18.543875 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:18.543883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:18.543901 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:18.544435 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:18.544895 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:18.544900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:18.544903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:18.545170 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:18.545796 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:18.545804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:18.546258 (+ 454us) client_negotiation.cc:769] Sending connection context
0924 11:50:18.547575 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0924 11:50:18.548140 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36269,"thread_start_us":37921,"threads_started":1}
I20260924 11:50:18.551646 14394 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37511
I20260924 11:50:18.552019 14394 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:18.552865 14394 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:18.555393 14180 ts_manager.cc:194] Registered new tserver with Master: 80958090704b467b8affe0c4b0cb45ad (127.1.210.1:36347)
I20260924 11:50:18.557449 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:53929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:50:18.584101 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37511
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:50:18 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250617, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
W20260924 11:50:19.075230 14400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:19.075557 14400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:19.075736 14400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:19.083403 14400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:19.083573 14400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:19.083772 14400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:50:19.094087 14400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.210.20:45345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:19.095511 14400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:19.096892 14400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:19.111138 14414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:19.111138 14415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:19.112041 14417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:19.112565 14400 server_base.cc:1034] running on GCE node
I20260924 11:50:19.113338 14400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:19.114382 14400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:19.115670 14400 hybrid_clock.cc:648] HybridClock initialized: now 1790250619115611 us; error 53 us; skew 500 ppm
I20260924 11:50:19.115976 14400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:19.119107 14400 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:50:19.120671 14400 webserver.cc:460] Webserver started at http://127.1.210.2:46227/ using document root <none> and password file <none>
I20260924 11:50:19.121414 14400 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:19.121539 14400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:19.121851 14400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:19.139535 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "c4310e5f07d44c9ca801ebd88454851a"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "96e6b8081574ab95e8f1dd50b7762381"
server_key_iv: "9423c26943d7011899cee2d7aaa4f202"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:19.140642 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "c4310e5f07d44c9ca801ebd88454851a"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "96e6b8081574ab95e8f1dd50b7762381"
server_key_iv: "9423c26943d7011899cee2d7aaa4f202"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:19.147414 14400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260924 11:50:19.164846 14424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:19.166738 14400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:50:19.166919 14400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "c4310e5f07d44c9ca801ebd88454851a"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "96e6b8081574ab95e8f1dd50b7762381"
server_key_iv: "9423c26943d7011899cee2d7aaa4f202"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:19.167236 14400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:19.202474 14400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:19.207542 14400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:19.207872 14400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:19.209024 14400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:19.210392 14400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:19.210461 14400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:19.210599 14400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:19.210647 14400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:19.251099 14400 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:36521
I20260924 11:50:19.251121 14536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:36521 every 8 connection(s)
I20260924 11:50:19.252473 14400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:50:19.256700 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14400
I20260924 11:50:19.256887 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:50:19.257543 1864 external_mini_cluster.cc:1428] Setting key bccc92223f5e81bfc2dbf77a9d5c09ab
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:19.276412 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:19.255304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57847 (local address 127.1.210.62:37511)
0924 11:50:19.255770 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:19.255775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:19.257357 (+ 1582us) server_negotiation.cc:407] Connection header received
0924 11:50:19.259225 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:19.259231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:19.259333 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:19.259434 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:19.261520 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.262751 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.265422 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.266139 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.266265 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:19.270223 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:19.270229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:19.270232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:19.270271 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:19.273187 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.274071 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.274075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.274077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.274194 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.274644 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.274649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.274654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.274960 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:19.275133 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:19.275812 (+ 679us) server_negotiation.cc:299] Negotiation successful
0924 11:50:19.276034 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
I20260924 11:50:19.277556 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:19.255932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37511 (local address 127.1.210.2:57847)
0924 11:50:19.256854 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:19.256911 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:19.258969 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:19.259859 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:19.259871 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:19.260438 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:19.261303 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:19.261332 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.262944 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.262952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:19.265251 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:19.265268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.266563 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.266572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:19.266911 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:19.267709 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:19.267743 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:19.269997 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:19.273331 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:19.273339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:19.273355 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:19.273897 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:19.274311 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:19.274317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:19.274321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:19.274522 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:19.275110 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:19.275117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:19.275456 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 11:50:19.275715 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:50:19.276229 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":394,"threads_started":1}
I20260924 11:50:19.279049 14537 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37511
I20260924 11:50:19.279371 14537 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:19.280148 14537 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:19.281594 14180 ts_manager.cc:194] Registered new tserver with Master: c4310e5f07d44c9ca801ebd88454851a (127.1.210.2:36521)
I20260924 11:50:19.282377 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:57847
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:50:19.318504 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37511
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:19.561183 14394 heartbeater.cc:503] Master 127.1.210.62:37511 was elected leader, sending a full tablet report...
W20260924 11:50:19.665007 14543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:19.665278 14543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:19.665434 14543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:19.672605 14543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:19.672744 14543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:19.672876 14543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:50:19.682226 14543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.210.20:45345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:19.683339 14543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:19.684489 14543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:19.697297 14549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:19.697278 14551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:19.697273 14548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:19.698204 14543 server_base.cc:1034] running on GCE node
I20260924 11:50:19.698755 14543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:19.699802 14543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:19.701051 14543 hybrid_clock.cc:648] HybridClock initialized: now 1790250619700978 us; error 84 us; skew 500 ppm
I20260924 11:50:19.701391 14543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:19.704808 14543 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:50:19.708581 14543 webserver.cc:460] Webserver started at http://127.1.210.3:38949/ using document root <none> and password file <none>
I20260924 11:50:19.709343 14543 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:19.709501 14543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:19.709816 14543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:19.724344 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "635bb4b00a39462fbc819258e9be0cdf"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "e5edf38d0fbe186975b901f0797e2274"
server_key_iv: "5656e67d4ea41d2d0829b9607713030b"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:19.725349 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "635bb4b00a39462fbc819258e9be0cdf"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "e5edf38d0fbe186975b901f0797e2274"
server_key_iv: "5656e67d4ea41d2d0829b9607713030b"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:19.730717 14543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:50:19.748034 14558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:19.749684 14543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:50:19.749888 14543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "635bb4b00a39462fbc819258e9be0cdf"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "e5edf38d0fbe186975b901f0797e2274"
server_key_iv: "5656e67d4ea41d2d0829b9607713030b"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:19.750113 14543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:19.766580 14543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:19.770185 14543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:19.770475 14543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:19.771370 14543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:19.772974 14543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:19.773088 14543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:19.773219 14543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:19.773306 14543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:19.806255 14543 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:36449
I20260924 11:50:19.806293 14670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:36449 every 8 connection(s)
I20260924 11:50:19.807677 14543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:50:19.813547 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14543
I20260924 11:50:19.813732 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:50:19.814159 1864 external_mini_cluster.cc:1428] Setting key cfc7d9a7259432435f932bda5354085e
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:19.827127 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:19.810529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:49479 (local address 127.1.210.62:37511)
0924 11:50:19.810984 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:19.810988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:19.811786 (+ 798us) server_negotiation.cc:407] Connection header received
0924 11:50:19.813587 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:19.813592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:19.813723 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:19.813855 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:19.815279 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.815999 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.817600 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.818036 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.818175 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:19.821487 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:19.821492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:19.821495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:19.821536 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:19.824098 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.824816 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.824826 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.824828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.824965 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.825462 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.825468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.825476 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.825724 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:19.825888 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:19.826551 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 11:50:19.826787 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":254,"threads_started":1}
I20260924 11:50:19.828233 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:19.810941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37511 (local address 127.1.210.3:49479)
0924 11:50:19.811616 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:19.811659 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:19.813329 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:19.814096 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:19.814104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:19.814544 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:19.815105 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:19.815123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.816142 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.816149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:19.817463 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:19.817474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.818280 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.818288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:19.818508 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:19.819176 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:19.819197 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:19.821279 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:19.824247 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:19.824255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:19.824272 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:19.824681 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:19.825133 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:19.825138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:19.825141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:19.825338 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:19.825867 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:19.825874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:19.826186 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 11:50:19.826448 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 11:50:19.826929 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":298,"threads_started":1}
I20260924 11:50:19.829799 14671 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37511
I20260924 11:50:19.830103 14671 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:19.830843 14671 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:19.832208 14179 ts_manager.cc:194] Registered new tserver with Master: 635bb4b00a39462fbc819258e9be0cdf (127.1.210.3:36449)
I20260924 11:50:19.833158 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:49479
I20260924 11:50:19.841482 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:50:19.843106 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14261
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:50:19.858510 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:19.843971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.1.210.1:36347)
0924 11:50:19.844601 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:19.844607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:19.844631 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:50:19.844685 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:19.844691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:19.844877 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:19.845002 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:19.846408 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.847382 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.848762 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.849363 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.849459 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:19.853384 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:19.853394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:19.853406 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:19.853441 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:19.856103 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.856787 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.856793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.856796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.856935 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.857278 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.857281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.857284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.857596 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:19.857817 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:19.858037 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 11:50:19.858241 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":333,"threads_started":1}
I20260924 11:50:19.859750 14389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:19.942117 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14261
I20260924 11:50:19.957661 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14400
Sep 24 11:50:19 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:50:19.973039 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:19.958569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.1.210.2:36521)
0924 11:50:19.959126 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:19.959132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:19.959157 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:50:19.959215 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:19.959222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:19.959435 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:19.959558 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:19.960803 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.961821 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.963014 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:19.963544 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:19.963635 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:19.967647 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:19.967657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:19.967670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:19.967707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:19.970328 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.971002 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.971008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.971011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.971134 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:19.971483 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:19.971486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:19.971488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:19.971847 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:19.972050 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:19.972353 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:50:19.972548 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":295,"threads_started":1}
I20260924 11:50:19.974501 14532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:20.055274 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14400
I20260924 11:50:20.070636 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14543
Sep 24 11:50:20 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:50:20.087560 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:20.071636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.1.210.3:36449)
0924 11:50:20.072245 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:20.072252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:20.072280 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:50:20.072337 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:20.072343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:20.072551 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:20.072719 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:20.074523 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:20.075584 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:20.076890 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:20.077504 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:20.077649 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:20.081928 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:20.081938 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:20.081952 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:20.081987 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:20.084571 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:20.085444 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:20.085457 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:20.085459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:20.085589 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:20.085975 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:20.085979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:20.085981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:20.086324 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:20.086648 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:20.086951 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:50:20.087211 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":327,"threads_started":1}
I20260924 11:50:20.088817 14666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:20.171684 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14543
I20260924 11:50:20.187435 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14148
I20260924 11:50:20.188674 14210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:20.293498 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14148
I20260924 11:50:20.317416 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37511
--webserver_interface=127.1.210.62
--webserver_port=35643
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:37511
--ranger_config_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:20.630270 14683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:20.630554 14683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:20.630754 14683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:20.637758 14683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:20.637850 14683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:20.637949 14683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:20.638053 14683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:20.638137 14683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:20.647817 14683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:37511
--ranger_config_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:20.649052 14683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:20.650365 14683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:20.662052 14688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:20.662058 14689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:20.661952 14691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:20.663331 14683 server_base.cc:1034] running on GCE node
I20260924 11:50:20.664022 14683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:20.665447 14683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:20.666644 14683 hybrid_clock.cc:648] HybridClock initialized: now 1790250620666613 us; error 28 us; skew 500 ppm
I20260924 11:50:20.666988 14683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:20 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:20.669904 14683 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:20.671361 14683 webserver.cc:460] Webserver started at http://127.1.210.62:35643/ using document root <none> and password file <none>
I20260924 11:50:20.671991 14683 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:20.672134 14683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:50:20 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:20.738998 14683 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:50:20.742050 14699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:20.743392 14683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:50:20.743604 14683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "cdeb351abefd42c09e465c7d5c02d718"
format_stamp: "Formatted at 2026-09-24 11:50:16 on dist-test-slave-dd32"
server_key: "fd38b91e10e4e2beb46f886d56002657"
server_key_iv: "75acd0c26660a5e157fd2f14d29d6ff6"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:20.744536 14683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:20.758617 14683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:20.762042 14683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:20.762313 14683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:20.780253 14683 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:37511
I20260924 11:50:20.780277 14750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:37511 every 8 connection(s)
I20260924 11:50:20.781857 14683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:20.788440 14751 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:50:20.789521 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14683
I20260924 11:50:20.790320 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:36347
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:37511
--webserver_port=33277
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:20.791792 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Bootstrap starting.
I20260924 11:50:20.816587 14751 log.cc:824] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:20.827494 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:50:20.828089 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Bootstrap complete.
I20260924 11:50:20.840727 14751 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:20.841894 14751 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdeb351abefd42c09e465c7d5c02d718, State: Initialized, Role: FOLLOWER
I20260924 11:50:20.842578 14751 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:20.842836 14751 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:20.842950 14751 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:20.843122 14751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:50:20.848927 14751 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:20.849576 14751 leader_election.cc:302] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdeb351abefd42c09e465c7d5c02d718; no voters:
I20260924 11:50:20.850235 14751 leader_election.cc:288] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:50:20.850391 14755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:50:20.851711 14755 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [term 2 LEADER]: Becoming Leader. State: Replica: cdeb351abefd42c09e465c7d5c02d718, State: Running, Role: LEADER
I20260924 11:50:20.852368 14751 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:20.852268 14755 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } }
I20260924 11:50:20.855130 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdeb351abefd42c09e465c7d5c02d718. Latest consensus state: current_term: 2 leader_uuid: "cdeb351abefd42c09e465c7d5c02d718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } } }
I20260924 11:50:20.855562 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:20.856101 14756 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cdeb351abefd42c09e465c7d5c02d718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeb351abefd42c09e465c7d5c02d718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37511 } } }
I20260924 11:50:20.856678 14756 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:20.858500 14761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:20.864262 14761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:20.865628 14761 catalog_manager.cc:1259] Loaded cluster ID: 1ed4de45a96f46bcab223ae3234d7719
I20260924 11:50:20.865782 14761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:20.869690 14761 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:20.871153 14761 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Loaded TSK: 0
I20260924 11:50:20.872646 14761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:50:20.882292 14751 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 11:50:21.245407 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:21.245937 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:21.246156 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:21.258281 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:21.258457 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:21.258689 14753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:50:21.274861 14753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:36347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=33277
--tserver_master_addrs=127.1.210.62:37511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:21.276842 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:21.278744 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:21.302338 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:21.303098 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:21.315873 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:21.317613 14753 server_base.cc:1034] running on GCE node
I20260924 11:50:21.318630 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:21.323331 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:21.326938 14753 hybrid_clock.cc:648] HybridClock initialized: now 1790250621326835 us; error 85 us; skew 500 ppm
I20260924 11:50:21.327452 14753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:21 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:21.332141 14753 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:50:21.337468 14753 webserver.cc:460] Webserver started at http://127.1.210.1:33277/ using document root <none> and password file <none>
I20260924 11:50:21.338485 14753 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:21.338662 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:50:21 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:21.406898 14753 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.006s sys 0.003s
I20260924 11:50:21.414521 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:21.416708 14753 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.004s
I20260924 11:50:21.416924 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "80958090704b467b8affe0c4b0cb45ad"
format_stamp: "Formatted at 2026-09-24 11:50:18 on dist-test-slave-dd32"
server_key: "af3a0196778c3745f249387fed73ef46"
server_key_iv: "f3f52526ca6b79e11bbb82d07872302c"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:21.418282 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:21.448426 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:21.453003 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:21.453326 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:21.454504 14753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:21.456097 14753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:21.456218 14753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:21.456344 14753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:21.456444 14753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:21.508422 14753 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:36347
I20260924 11:50:21.508540 14912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:36347 every 8 connection(s)
I20260924 11:50:21.510555 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:50:21.516068 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14753
I20260924 11:50:21.516898 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:36521
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:37511
--webserver_port=46227
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:50:21 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:21.592262 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:21.515658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:51979 (local address 127.1.210.62:37511)
0924 11:50:21.551704 (+ 36046us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:21.551727 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:21.551784 (+ 57us) server_negotiation.cc:407] Connection header received
0924 11:50:21.552269 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:21.552290 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:21.552828 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:21.553310 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:21.555903 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:21.557557 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:21.560485 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:21.561823 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:21.562049 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:21.576687 (+ 14638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:21.576702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:21.576726 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:21.576788 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:21.584760 (+ 7972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:21.585962 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:21.585973 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:21.585981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:21.586210 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:21.586844 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:21.586849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:21.586852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:21.587496 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:21.587709 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:21.590497 (+ 2788us) server_negotiation.cc:299] Negotiation successful
0924 11:50:21.590919 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35715,"thread_start_us":40453,"threads_started":1}
I20260924 11:50:21.593065 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:21.514423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37511 (local address 127.1.210.1:51979)
0924 11:50:21.540797 (+ 26374us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:21.540866 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:21.550681 (+ 9815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:21.554078 (+ 3397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:21.554092 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:21.554791 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:21.555617 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:21.555645 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:21.557757 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:21.557766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:21.560223 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:21.560256 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:21.562511 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:21.562522 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:21.562899 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:21.571414 (+ 8515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:21.571457 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:21.576192 (+ 4735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:21.585012 (+ 8820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:21.585022 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:21.585037 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:21.585740 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:21.586388 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:21.586394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:21.586397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:21.586688 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:21.587803 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:21.587811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:21.588197 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 11:50:21.588516 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 11:50:21.589118 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25985,"thread_start_us":26548,"threads_started":1}
I20260924 11:50:21.597604 14913 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37511
I20260924 11:50:21.597954 14913 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:21.598892 14913 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
W20260924 11:50:21.602663 14913 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:37511 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:50:21.602957 14913 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:21.603269 14913 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:21.604756 14913 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:21.605146 14913 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
W20260924 11:50:21.606340 14913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:50:21.989248 14917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:21.989710 14917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:21.989905 14917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:22.003072 14917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:22.003240 14917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:22.003477 14917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:50:22.021335 14917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:36521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=46227
--tserver_master_addrs=127.1.210.62:37511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:22.023116 14917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:22.024991 14917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:22.047418 14931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:22.053056 14932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:22.054044 14934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:22.056749 14917 server_base.cc:1034] running on GCE node
I20260924 11:50:22.057627 14917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:22.066998 14917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:22.068306 14917 hybrid_clock.cc:648] HybridClock initialized: now 1790250622068234 us; error 72 us; skew 500 ppm
I20260924 11:50:22.068804 14917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:22.073532 14917 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:50:22.075577 14917 webserver.cc:460] Webserver started at http://127.1.210.2:46227/ using document root <none> and password file <none>
I20260924 11:50:22.076350 14917 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:22.076502 14917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:22.108626 14917 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:50:22.112489 14944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:22.114436 14917 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:50:22.114615 14917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "c4310e5f07d44c9ca801ebd88454851a"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "96e6b8081574ab95e8f1dd50b7762381"
server_key_iv: "9423c26943d7011899cee2d7aaa4f202"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:22.116036 14917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:22.137794 14917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:22.142931 14917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:22.143285 14917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:22.144603 14917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:22.146499 14917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:22.146667 14917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 11:50:22.146836 14917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:22.146925 14917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:22.231964 14917 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:36521
I20260924 11:50:22.231986 15056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:36521 every 8 connection(s)
I20260924 11:50:22.233742 14917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:50:22.235133 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 14917
I20260924 11:50:22.235937 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.20:45345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:36449
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:37511
--webserver_port=38949
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:22.260876 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:22.237864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:43385 (local address 127.1.210.62:37511)
0924 11:50:22.243377 (+ 5513us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:22.243385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:22.243409 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:50:22.243483 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:22.243490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:22.243578 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:22.243716 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:22.245804 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:22.246971 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:22.249600 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:22.250222 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:22.250362 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:22.254875 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:22.254880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:22.254883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:22.254916 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:22.257751 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:22.258439 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:22.258451 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:22.258455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:22.258588 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:22.259046 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:22.259055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:22.259058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:22.259321 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:22.259479 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:22.260268 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 11:50:22.260485 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5311,"thread_start_us":5208,"threads_started":1}
I20260924 11:50:22.262198 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:22.237292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37511 (local address 127.1.210.2:43385)
0924 11:50:22.238596 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:22.238645 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:22.241405 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:22.244031 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:22.244044 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:22.244680 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:22.245563 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:22.245591 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:22.247175 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:22.247180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:22.249408 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:22.249432 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:22.250579 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:22.250587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:22.250893 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:22.251721 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:22.251752 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:22.254604 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:22.257885 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:22.257894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:22.257910 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:22.258312 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:22.258709 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:22.258714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:22.258717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:22.258922 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:22.259448 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:22.259456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:22.259839 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 11:50:22.260468 (+ 629us) client_negotiation.cc:240] Negotiation successful
0924 11:50:22.261106 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"spinlock_wait_cycles":17280,"thread_start_us":780,"threads_started":1}
I20260924 11:50:22.263877 15057 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37511
I20260924 11:50:22.264222 15057 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:22.265116 15057 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:22.267251 14716 ts_manager.cc:194] Registered new tserver with Master: c4310e5f07d44c9ca801ebd88454851a (127.1.210.2:36521)
I20260924 11:50:22.269475 14716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:43385
I20260924 11:50:22.606489 14913 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:22.606961 14913 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:22.608659 14716 ts_manager.cc:194] Registered new tserver with Master: 80958090704b467b8affe0c4b0cb45ad (127.1.210.1:36347)
I20260924 11:50:22.610008 14716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:51979
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
W20260924 11:50:22.675928 15061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:22.676409 15061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:22.676605 15061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:22.687517 15061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:22.687712 15061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:22.687913 15061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:50:22.701560 15061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.20:45345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:36449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=38949
--tserver_master_addrs=127.1.210.62:37511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:22.703558 15061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:22.705431 15061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:22.736042 15067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:22.740697 15068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:22.751709 15070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:22.752466 15061 server_base.cc:1034] running on GCE node
I20260924 11:50:22.753371 15061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:22.754712 15061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:22.756018 15061 hybrid_clock.cc:648] HybridClock initialized: now 1790250622755955 us; error 49 us; skew 500 ppm
I20260924 11:50:22.756392 15061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:22.760705 15061 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:50:22.798532 15061 webserver.cc:460] Webserver started at http://127.1.210.3:38949/ using document root <none> and password file <none>
I20260924 11:50:22.799782 15061 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:22.799981 15061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:22.827605 15061 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:50:22.831408 15078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:22.833341 15061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 11:50:22.833524 15061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "635bb4b00a39462fbc819258e9be0cdf"
format_stamp: "Formatted at 2026-09-24 11:50:19 on dist-test-slave-dd32"
server_key: "e5edf38d0fbe186975b901f0797e2274"
server_key_iv: "5656e67d4ea41d2d0829b9607713030b"
server_key_version: "kuduclusterkey@0"
I20260924 11:50:22.834820 15061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:22.859365 15061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:22.864588 15061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:22.864914 15061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:22.867053 15061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:22.868793 15061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:22.868902 15061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:22.869050 15061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:22.869168 15061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:50:22 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for HTTP/127.1.210.20@KRBTEST.COM
I20260924 11:50:23.053181 15061 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:36449
I20260924 11:50:23.053313 15191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:36449 every 8 connection(s)
I20260924 11:50:23.054939 15061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:50:23.062991 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15061
Sep 24 11:50:23 dist-test-slave-dd32 krb5kdc[13704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:23.092823 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.058437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37511 (local address 127.1.210.3:52789)
0924 11:50:23.060774 (+ 2337us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.060826 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:23.069146 (+ 8320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:23.072073 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:23.072085 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:23.072698 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:23.073477 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.073499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.075161 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.075167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.076912 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.076934 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.078257 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.078264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.078538 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.079935 (+ 1397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:23.079968 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:23.083542 (+ 3574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:23.087533 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:23.087542 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:23.087561 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:23.088156 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:23.088607 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:23.088612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:23.088616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:23.088868 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:23.090139 (+ 1271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:23.090147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:23.090574 (+ 427us) client_negotiation.cc:769] Sending connection context
0924 11:50:23.090867 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 11:50:23.091437 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"thread_start_us":1834,"threads_started":1}
I20260924 11:50:23.093924 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.059023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:52789 (local address 127.1.210.62:37511)
0924 11:50:23.063935 (+ 4912us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.063941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.063964 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:50:23.069541 (+ 5577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.069547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.069618 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.069713 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:23.073699 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.074952 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.077108 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.077817 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.077907 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.083812 (+ 5905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:23.083821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:23.083825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:23.083879 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:23.087367 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.088314 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.088319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.088323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.088473 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.089003 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.089008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.089011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.089401 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:23.089551 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.093347 (+ 3796us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.093566 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4792,"thread_start_us":6088,"threads_started":1}
I20260924 11:50:23.096892 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.064450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.1.210.62:37511)
0924 11:50:23.066152 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.066158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.066176 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:50:23.066239 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.066244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.066337 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.066448 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:23.068556 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.069714 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.071812 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.072487 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.072609 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.076441 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:23.076447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:23.076452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:23.076488 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:23.082733 (+ 6245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.083882 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.083887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.083891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.089646 (+ 5755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.094444 (+ 4798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.094449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.094453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.094780 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:23.095004 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.096328 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.096547 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":1459,"threads_started":1}
I20260924 11:50:23.100994 15192 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37511
I20260924 11:50:23.101328 15192 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:23.102140 15192 heartbeater.cc:511] Master 127.1.210.62:37511 requested a full tablet report, sending...
I20260924 11:50:23.103850 14715 ts_manager.cc:194] Registered new tserver with Master: 635bb4b00a39462fbc819258e9be0cdf (127.1.210.3:36449)
I20260924 11:50:23.105170 14715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:52789
I20260924 11:50:23.112682 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:50:23.154099 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.137138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.1.210.62:37511)
0924 11:50:23.137334 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.137342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.137417 (+ 75us) server_negotiation.cc:407] Connection header received
0924 11:50:23.137724 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.137731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.137808 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.137918 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:23.139554 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.140672 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.142706 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.143391 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.143520 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.147134 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:23.147139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:23.147143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:23.147176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:23.150750 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.151547 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.151552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.151556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.151694 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.152129 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.152134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.152137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.152534 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:23.152834 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.153514 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.153742 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:50:23.159799 14715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:50:23.171315 14715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:50:23.271724 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.258480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:36521 (local address 127.0.0.1:55530)
0924 11:50:23.261853 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.261876 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:23.262012 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:23.264278 (+ 2266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:23.264283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:23.264306 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:23.264851 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.264866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.266788 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.266794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.268837 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.268852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.270893 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.270898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.271075 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.271110 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:50:23.271197 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:50:23.271318 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"mutex_wait_us":25,"thread_start_us":5876,"threads_started":1}
I20260924 11:50:23.272631 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.258795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.1.210.2:36521)
0924 11:50:23.263777 (+ 4982us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.263783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.263804 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:50:23.263873 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.263878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.264105 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.264284 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:23.265066 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.266538 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.269020 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.270406 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.270524 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.270762 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.272118 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.272281 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4834,"thread_start_us":4712,"threads_started":1}
I20260924 11:50:23.272776 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.208000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58232)
0924 11:50:23.250415 (+ 42415us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.250460 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:23.250651 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:23.260987 (+ 10336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:23.260993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:23.261018 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:23.261368 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.261381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.263432 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.263437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.265825 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.265839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.268026 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.268033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.268236 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.268296 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 11:50:23.268424 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 11:50:23.268646 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42131,"mutex_wait_us":165,"thread_start_us":50053,"threads_started":1}
I20260924 11:50:23.273741 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.208441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:36449 (local address 127.0.0.1:45702)
0924 11:50:23.251907 (+ 43466us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.251925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:23.252040 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:23.253259 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:23.253266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:23.253298 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:23.253706 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.253720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.255823 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.255828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.269299 (+ 13471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.269314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.272958 (+ 3644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.272965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.273200 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.273233 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:50:23.273299 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 11:50:23.273457 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43355,"thread_start_us":42478,"threads_started":1}
I20260924 11:50:23.273903 15057 heartbeater.cc:503] Master 127.1.210.62:37511 was elected leader, sending a full tablet report...
I20260924 11:50:23.274463 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.208913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.1.210.3:36449)
0924 11:50:23.249026 (+ 40113us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.249035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.252175 (+ 3140us) server_negotiation.cc:407] Connection header received
0924 11:50:23.252276 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.252284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.252573 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.252779 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:23.253974 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.255656 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.269501 (+ 13845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.272178 (+ 2677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.272307 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.272504 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.273961 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.274133 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39916,"thread_start_us":40178,"threads_started":1}
I20260924 11:50:23.274770 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.208036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.1.210.1:36347)
0924 11:50:23.259065 (+ 51029us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.259074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.259115 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:50:23.260438 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.260452 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.260765 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.261009 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:23.261590 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.263241 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.266008 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.267462 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.267596 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.267833 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.274244 (+ 6411us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.274421 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50847,"thread_start_us":52112,"threads_started":1}
I20260924 11:50:23.277503 14992 tablet_service.cc:1467] Processing CreateTablet for tablet efa92afa24e7471fa94560f2ce6456bd (DEFAULT_TABLE table=test-table [id=02803126cc59476880d30e36c2f233a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:23.278038 15126 tablet_service.cc:1467] Processing CreateTablet for tablet efa92afa24e7471fa94560f2ce6456bd (DEFAULT_TABLE table=test-table [id=02803126cc59476880d30e36c2f233a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:23.278173 14848 tablet_service.cc:1467] Processing CreateTablet for tablet efa92afa24e7471fa94560f2ce6456bd (DEFAULT_TABLE table=test-table [id=02803126cc59476880d30e36c2f233a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:23.279198 14992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa92afa24e7471fa94560f2ce6456bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:23.279369 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa92afa24e7471fa94560f2ce6456bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:23.279449 14848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa92afa24e7471fa94560f2ce6456bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:23.301697 15218 tablet_bootstrap.cc:492] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: Bootstrap starting.
I20260924 11:50:23.302841 15221 tablet_bootstrap.cc:492] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Bootstrap starting.
I20260924 11:50:23.303438 15217 tablet_bootstrap.cc:492] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Bootstrap starting.
I20260924 11:50:23.305624 15218 tablet_bootstrap.cc:654] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:23.306915 15217 tablet_bootstrap.cc:654] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:23.306949 15218 log.cc:824] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:23.308239 15217 log.cc:824] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:23.308866 15221 tablet_bootstrap.cc:654] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:23.310082 15221 log.cc:824] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:23.315510 15217 tablet_bootstrap.cc:492] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: No bootstrap required, opened a new log
I20260924 11:50:23.315923 15217 ts_tablet_manager.cc:1397] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260924 11:50:23.324213 15221 tablet_bootstrap.cc:492] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: No bootstrap required, opened a new log
I20260924 11:50:23.324236 15218 tablet_bootstrap.cc:492] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: No bootstrap required, opened a new log
I20260924 11:50:23.324548 15221 ts_tablet_manager.cc:1397] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20260924 11:50:23.324546 15218 ts_tablet_manager.cc:1397] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.005s
I20260924 11:50:23.328780 15217 raft_consensus.cc:348] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.329291 15217 raft_consensus.cc:374] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:23.329425 15217 raft_consensus.cc:729] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4310e5f07d44c9ca801ebd88454851a, State: Initialized, Role: FOLLOWER
I20260924 11:50:23.332020 15217 consensus_queue.cc:260] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.335333 15221 raft_consensus.cc:348] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.335503 15218 raft_consensus.cc:348] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.335815 15221 raft_consensus.cc:374] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:23.335914 15221 raft_consensus.cc:729] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635bb4b00a39462fbc819258e9be0cdf, State: Initialized, Role: FOLLOWER
I20260924 11:50:23.335997 15218 raft_consensus.cc:374] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:23.336148 15218 raft_consensus.cc:729] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80958090704b467b8affe0c4b0cb45ad, State: Initialized, Role: FOLLOWER
I20260924 11:50:23.336621 15221 consensus_queue.cc:260] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.337041 15218 consensus_queue.cc:260] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.340210 15217 ts_tablet_manager.cc:1428] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Time spent starting tablet: real 0.024s user 0.010s sys 0.010s
I20260924 11:50:23.341588 15192 heartbeater.cc:503] Master 127.1.210.62:37511 was elected leader, sending a full tablet report...
I20260924 11:50:23.341791 15221 ts_tablet_manager.cc:1428] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260924 11:50:23.342555 15218 ts_tablet_manager.cc:1428] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: Time spent starting tablet: real 0.018s user 0.005s sys 0.011s
I20260924 11:50:23.342917 14913 heartbeater.cc:503] Master 127.1.210.62:37511 was elected leader, sending a full tablet report...
I20260924 11:50:23.359179 15228 raft_consensus.cc:482] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:23.359501 15228 raft_consensus.cc:504] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.361756 15228 leader_election.cc:288] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 635bb4b00a39462fbc819258e9be0cdf (127.1.210.3:36449), c4310e5f07d44c9ca801ebd88454851a (127.1.210.2:36521)
I20260924 11:50:23.369946 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.362095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:58559 (local address 127.1.210.3:36449)
0924 11:50:23.362302 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.362309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.363771 (+ 1462us) server_negotiation.cc:407] Connection header received
0924 11:50:23.363986 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.363992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.364066 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.364151 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:23.364894 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.365962 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.367940 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.368878 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.369004 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.369128 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.369550 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.369691 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:50:23.370056 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.362628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:36449 (local address 127.1.210.1:58559)
0924 11:50:23.363687 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.363703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:23.363858 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:23.364216 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:23.364223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:23.364311 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:23.364707 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.364719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.366100 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.366105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.367786 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.367798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.369139 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.369148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.369368 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.369401 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:50:23.369502 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:50:23.369628 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":29,"spinlock_wait_cycles":21888,"thread_start_us":706,"threads_started":1}
I20260924 11:50:23.370712 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.362176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:35669 (local address 127.1.210.2:36521)
0924 11:50:23.362363 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.362368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.363539 (+ 1171us) server_negotiation.cc:407] Connection header received
0924 11:50:23.363851 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.363856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.363931 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.364031 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:23.364636 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.365772 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.368499 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.369593 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.369713 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.369819 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.370352 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.370481 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:50:23.371295 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.362626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:36521 (local address 127.1.210.1:35669)
0924 11:50:23.363421 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.363444 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:23.363653 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:23.364042 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:23.364048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:23.364071 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:23.364491 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.364500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.365943 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.365949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.368330 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:23.368344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.369982 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.369989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:23.370172 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.370204 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:50:23.370916 (+ 712us) client_negotiation.cc:240] Negotiation successful
0924 11:50:23.371050 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":163,"thread_start_us":409,"threads_started":1}
I20260924 11:50:23.371120 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa92afa24e7471fa94560f2ce6456bd" candidate_uuid: "80958090704b467b8affe0c4b0cb45ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635bb4b00a39462fbc819258e9be0cdf" is_pre_election: true
I20260924 11:50:23.371650 15146 raft_consensus.cc:2454] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80958090704b467b8affe0c4b0cb45ad in term 0.
I20260924 11:50:23.372304 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa92afa24e7471fa94560f2ce6456bd" candidate_uuid: "80958090704b467b8affe0c4b0cb45ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4310e5f07d44c9ca801ebd88454851a" is_pre_election: true
I20260924 11:50:23.372771 15012 raft_consensus.cc:2454] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80958090704b467b8affe0c4b0cb45ad in term 0.
I20260924 11:50:23.373510 14803 leader_election.cc:302] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 635bb4b00a39462fbc819258e9be0cdf, 80958090704b467b8affe0c4b0cb45ad, c4310e5f07d44c9ca801ebd88454851a; no voters:
I20260924 11:50:23.373944 15228 raft_consensus.cc:2781] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:50:23.374094 15228 raft_consensus.cc:482] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:50:23.374214 15228 raft_consensus.cc:3037] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:23.376288 15228 raft_consensus.cc:504] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.376916 15228 leader_election.cc:288] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [CANDIDATE]: Term 1 election: Requested vote from peers 635bb4b00a39462fbc819258e9be0cdf (127.1.210.3:36449), c4310e5f07d44c9ca801ebd88454851a (127.1.210.2:36521)
I20260924 11:50:23.377311 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa92afa24e7471fa94560f2ce6456bd" candidate_uuid: "80958090704b467b8affe0c4b0cb45ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635bb4b00a39462fbc819258e9be0cdf"
I20260924 11:50:23.377424 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa92afa24e7471fa94560f2ce6456bd" candidate_uuid: "80958090704b467b8affe0c4b0cb45ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4310e5f07d44c9ca801ebd88454851a"
I20260924 11:50:23.377521 15146 raft_consensus.cc:3037] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:23.377609 15012 raft_consensus.cc:3037] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:23.380688 15146 raft_consensus.cc:2454] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80958090704b467b8affe0c4b0cb45ad in term 1.
I20260924 11:50:23.380688 15012 raft_consensus.cc:2454] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80958090704b467b8affe0c4b0cb45ad in term 1.
I20260924 11:50:23.381095 14802 leader_election.cc:302] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635bb4b00a39462fbc819258e9be0cdf, 80958090704b467b8affe0c4b0cb45ad; no voters:
I20260924 11:50:23.381489 15228 raft_consensus.cc:2781] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:23.382042 15228 raft_consensus.cc:686] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 1 LEADER]: Becoming Leader. State: Replica: 80958090704b467b8affe0c4b0cb45ad, State: Running, Role: LEADER
I20260924 11:50:23.382589 15228 consensus_queue.cc:237] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } }
I20260924 11:50:23.387451 14714 catalog_manager.cc:5495] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 80958090704b467b8affe0c4b0cb45ad (127.1.210.1). New cstate: current_term: 1 leader_uuid: "80958090704b467b8affe0c4b0cb45ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80958090704b467b8affe0c4b0cb45ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 } health_report { overall_health: UNKNOWN } } }
I20260924 11:50:23.411427 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.405100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.1.210.1:36347)
0924 11:50:23.405317 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.405324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.405451 (+ 127us) server_negotiation.cc:407] Connection header received
0924 11:50:23.405634 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.405639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.405742 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.405905 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:50:23.406273 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.407434 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.408827 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.409529 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.409654 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.410108 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:50:23.410862 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:50:23.411014 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.411063 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.411149 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 11:50:23.419696 15146 raft_consensus.cc:1261] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 1 FOLLOWER]: Refusing update from remote peer 80958090704b467b8affe0c4b0cb45ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:23.419723 15012 raft_consensus.cc:1261] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 1 FOLLOWER]: Refusing update from remote peer 80958090704b467b8affe0c4b0cb45ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:23.420765 15231 consensus_queue.cc:1035] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "635bb4b00a39462fbc819258e9be0cdf" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 36449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:23.421222 15228 consensus_queue.cc:1035] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4310e5f07d44c9ca801ebd88454851a" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:23.437770 15234 mvcc.cc:205] Tried to move back new op lower bound from 7332866553516752896 to 7332866553378926592. Current Snapshot: MvccSnapshot[applied={T|T < 7332866553516752896}]
I20260924 11:50:23.471071 14714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:50:23.471426 14714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:50:23.475859 14714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdeb351abefd42c09e465c7d5c02d718: Sending DeleteTablet for 3 replicas of tablet efa92afa24e7471fa94560f2ce6456bd
I20260924 11:50:23.476959 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet efa92afa24e7471fa94560f2ce6456bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:23 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:45702
I20260924 11:50:23.477020 14848 tablet_service.cc:1514] Processing DeleteTablet for tablet efa92afa24e7471fa94560f2ce6456bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:23 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:58232
I20260924 11:50:23.478726 14992 tablet_service.cc:1514] Processing DeleteTablet for tablet efa92afa24e7471fa94560f2ce6456bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:23 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:55530
I20260924 11:50:23.486238 15245 tablet_replica.cc:330] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: stopping tablet replica
I20260924 11:50:23.487010 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14753
I20260924 11:50:23.487341 15245 raft_consensus.cc:2229] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:50:23.489481 15243 tablet_replica.cc:330] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: stopping tablet replica
I20260924 11:50:23.490175 15243 raft_consensus.cc:2229] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:50:23.490733 15243 raft_consensus.cc:2258] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:23.491421 15245 raft_consensus.cc:2258] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:23.494040 15245 ts_tablet_manager.cc:1905] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:23.504143 15245 ts_tablet_manager.cc:1918] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:23.504356 15245 log.cc:1196] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/efa92afa24e7471fa94560f2ce6456bd
I20260924 11:50:23.504944 15245 ts_tablet_manager.cc:1939] T efa92afa24e7471fa94560f2ce6456bd P c4310e5f07d44c9ca801ebd88454851a: Deleting consensus metadata
I20260924 11:50:23.506778 14703 catalog_manager.cc:4841] TS c4310e5f07d44c9ca801ebd88454851a (127.1.210.2:36521): tablet efa92afa24e7471fa94560f2ce6456bd (table test-table [id=02803126cc59476880d30e36c2f233a1]) successfully deleted
I20260924 11:50:23.510361 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.487809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.1.210.1:36347)
0924 11:50:23.487991 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.487998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.488219 (+ 221us) server_negotiation.cc:407] Connection header received
0924 11:50:23.488454 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.488459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.488541 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.488703 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:23.490722 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.491892 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.494347 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.495057 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.495179 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.499741 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:23.499756 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:23.499773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:23.499812 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:23.504603 (+ 4791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.505898 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.505912 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.505918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.506143 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.506739 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.506749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.506755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.507362 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:23.507781 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.509369 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.509752 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:50:23.511224 14908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:23.513787 15243 ts_tablet_manager.cc:1905] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:23.522804 15243 ts_tablet_manager.cc:1918] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:23.522996 15243 log.cc:1196] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/efa92afa24e7471fa94560f2ce6456bd
I20260924 11:50:23.523610 15243 ts_tablet_manager.cc:1939] T efa92afa24e7471fa94560f2ce6456bd P 635bb4b00a39462fbc819258e9be0cdf: Deleting consensus metadata
I20260924 11:50:23.525331 14702 catalog_manager.cc:4841] TS 635bb4b00a39462fbc819258e9be0cdf (127.1.210.3:36449): tablet efa92afa24e7471fa94560f2ce6456bd (table test-table [id=02803126cc59476880d30e36c2f233a1]) successfully deleted
I20260924 11:50:23.631374 15244 tablet_replica.cc:330] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: stopping tablet replica
I20260924 11:50:23.631886 15244 raft_consensus.cc:2229] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:50:23.632583 15244 raft_consensus.cc:2258] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:23.634959 15244 ts_tablet_manager.cc:1905] T efa92afa24e7471fa94560f2ce6456bd P 80958090704b467b8affe0c4b0cb45ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:23.640446 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14753
I20260924 11:50:23.663932 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14917
W20260924 11:50:23.663844 14703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.1:36347 (error 108)
I20260924 11:50:23.666136 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.665681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58252)
0924 11:50:23.665878 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.665987 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 11:50:23.666397 14703 catalog_manager.cc:4584] TS 80958090704b467b8affe0c4b0cb45ad (127.1.210.1:36347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efa92afa24e7471fa94560f2ce6456bd: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
I20260924 11:50:23.682363 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.665238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.1.210.2:36521)
0924 11:50:23.665438 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.665443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.665459 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:50:23.665585 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.665590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.665665 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.665771 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:23.667302 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.668007 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.669402 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.669852 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.669984 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.676622 (+ 6638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:23.676635 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:23.676650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:23.676684 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:23.679665 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.680441 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.680448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.680451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.680576 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.680935 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.680938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.680940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.681336 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:23.681559 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.681871 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.682028 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:50:23.683074 15052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:23.720141 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.719554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58260)
0924 11:50:23.719803 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.719938 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 11:50:23.721333 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.720891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58276)
0924 11:50:23.721052 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.721170 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 11:50:23.775652 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14917
I20260924 11:50:23.781957 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.781446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58284)
0924 11:50:23.781709 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.781801 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260924 11:50:23.783149 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.782760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58298)
0924 11:50:23.782950 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.783017 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 11:50:23.794505 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15061
I20260924 11:50:23.807825 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.795296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.1.210.3:36449)
0924 11:50:23.795500 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:23.795505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:23.795526 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:50:23.795676 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:23.795682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:23.795767 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:23.795874 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:23.797114 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.797936 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.799239 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:23.799646 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:23.799779 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:23.802288 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:23.802311 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:23.802327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:23.802357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:23.804981 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.805677 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.805686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.805689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.805819 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:23.806299 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:23.806304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:23.806307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:23.806664 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:23.806975 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:23.807281 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 11:50:23.807491 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:50:23.808482 15187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:23.863781 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.863295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58312)
0924 11:50:23.863500 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.863604 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:50:23.864894 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:23.864491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:36347 (local address 127.0.0.1:58322)
0924 11:50:23.864697 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:23.864768 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 11:50:23.898376 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15061
I20260924 11:50:23.914657 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 14683
I20260924 11:50:23.915983 14746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:24.027724 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 14683
I20260924 11:50:24.049181 1864 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 11:50:24.384318 1864 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 11:50:24.384514 1864 mini_ranger.cc:66] Stopping Ranger...
I20260924 11:50:24.729367 1864 mini_ranger.cc:68] Stopped Ranger
2026-09-24 11:50:24.729 UTC [13734] LOG: received smart shutdown request
2026-09-24 11:50:24.732 UTC [13734] LOG: background worker "logical replication launcher" (PID 13743) exited with exit code 1
2026-09-24 11:50:24.732 UTC [13738] LOG: shutting down
2026-09-24 11:50:24.901 UTC [13734] LOG: database system is shut down
2026-09-24T11:50:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51753 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:50:25 dist-test-slave-dd32 krb5kdc[15259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:50:25 dist-test-slave-dd32 krb5kdc[15259](info): set up 2 sockets
Sep 24 11:50:25 dist-test-slave-dd32 krb5kdc[15259](info): commencing operation
krb5kdc: starting...
W20260924 11:50:26.145617 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:50:26 dist-test-slave-dd32 krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:50:26.368237 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:35699
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:35699,127.1.210.61:33087,127.1.210.60:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:26.693764 15272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:26.694099 15272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:26.694252 15272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:26.701511 15272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:26.701668 15272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:26.701767 15272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:26.701865 15272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:26.701937 15272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:26.711783 15272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:35699,127.1.210.61:33087,127.1.210.60:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:26.713166 15272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:26.714465 15272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:26.726037 15277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:26.726063 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:26.726018 15280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:26.727176 15272 server_base.cc:1034] running on GCE node
I20260924 11:50:26.727880 15272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:26.729241 15272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:26.730466 15272 hybrid_clock.cc:648] HybridClock initialized: now 1790250626730420 us; error 42 us; skew 500 ppm
I20260924 11:50:26.730783 15272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:26 dist-test-slave-dd32 krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:26.736990 15272 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:26.738490 15272 webserver.cc:460] Webserver started at http://127.1.210.62:34139/ using document root <none> and password file <none>
I20260924 11:50:26.739101 15272 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:26.739203 15272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:26.739437 15272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:26.742316 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "87bb8b8851b34293ac35dd1e8d3a1032"
format_stamp: "Formatted at 2026-09-24 11:50:26 on dist-test-slave-dd32"
server_key: "9c2a262fe53b6bf405e39d123f101704"
server_key_iv: "a0f61bf5d92c5b22cc1ca61ef72cd8d8"
server_key_version: "encryptionkey@0"
I20260924 11:50:26.743053 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "87bb8b8851b34293ac35dd1e8d3a1032"
format_stamp: "Formatted at 2026-09-24 11:50:26 on dist-test-slave-dd32"
server_key: "9c2a262fe53b6bf405e39d123f101704"
server_key_iv: "a0f61bf5d92c5b22cc1ca61ef72cd8d8"
server_key_version: "encryptionkey@0"
I20260924 11:50:26.748462 15272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:50:26.752426 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:26.753798 15272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:26.753968 15272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "87bb8b8851b34293ac35dd1e8d3a1032"
format_stamp: "Formatted at 2026-09-24 11:50:26 on dist-test-slave-dd32"
server_key: "9c2a262fe53b6bf405e39d123f101704"
server_key_iv: "a0f61bf5d92c5b22cc1ca61ef72cd8d8"
server_key_version: "encryptionkey@0"
I20260924 11:50:26.754154 15272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:26.766546 15272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:26.771221 15272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:26.771528 15272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:26.791162 15272 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:35699
I20260924 11:50:26.791165 15338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:35699 every 8 connection(s)
I20260924 11:50:26.792694 15272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:26.796535 15339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:26.798972 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15272
I20260924 11:50:26.799203 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:50:26.799971 1864 external_mini_cluster.cc:1428] Setting key b6000c05cf1141de2fc9b738153a3d2e
I20260924 11:50:26.805966 15339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.210.61@KRBTEST.COM; defaulting to no policy
Sep 24 11:50:26 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.61@KRBTEST.COM" created.
I20260924 11:50:26.830919 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.809925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:35699 (local address 127.0.0.1:37904)
0924 11:50:26.811179 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.811234 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:26.812960 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:26.813970 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:26.813975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:26.814532 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:26.814856 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:26.814872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:26.817187 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:26.817191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:26.818393 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:26.818405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:26.819855 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:26.819861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:26.820064 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:26.820756 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:26.820781 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:26.823097 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:26.826934 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:26.826944 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:26.826954 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:26.827363 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:26.827816 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:26.827820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:26.827822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:26.827972 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:26.828965 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:26.828971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:26.829239 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 11:50:26.829456 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 11:50:26.829843 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":61,"spinlock_wait_cycles":63744,"thread_start_us":768,"threads_started":1}
I20260924 11:50:26.830964 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.809925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.1.210.62:35699)
0924 11:50:26.810795 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:26.810811 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:26.811389 (+ 578us) server_negotiation.cc:407] Connection header received
0924 11:50:26.813390 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:26.813401 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:26.813726 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:26.813943 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:26.815050 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:26.816997 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:26.818546 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:26.819618 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:26.819850 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:26.823373 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:26.823383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:26.823395 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:26.823424 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:26.826787 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:26.827496 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:26.827508 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:26.827515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:26.827706 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:26.828082 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:26.828090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:26.828096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:26.828830 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:26.829048 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:26.829638 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 11:50:26.830032 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
I20260924 11:50:26.833817 15339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:50:26.834854 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.834387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50600)
0924 11:50:26.834581 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.834709 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:50:26.835100 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
I20260924 11:50:26.836772 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.836313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50602)
0924 11:50:26.836517 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.836605 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:50:26.836988 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:33087: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab.
Entry for principal kudu/127.1.210.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab.
I20260924 11:50:26.854005 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:33087
--webserver_interface=127.1.210.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab
--principal=kudu/127.1.210.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:35699,127.1.210.61:33087,127.1.210.60:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:26.886372 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } attempt: 1
I20260924 11:50:26.887668 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.887150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50610)
0924 11:50:26.887386 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.887509 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 11:50:26.888716 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.888355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50612)
0924 11:50:26.888510 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.888581 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 11:50:26.889029 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:33087: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
I20260924 11:50:26.957432 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } attempt: 2
I20260924 11:50:26.958689 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.958180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50614)
0924 11:50:26.958443 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.958557 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:50:26.959690 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:26.959375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50616)
0924 11:50:26.959515 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:26.959575 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:50:26.959947 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:33087: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
I20260924 11:50:27.051339 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } attempt: 3
I20260924 11:50:27.052587 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.052104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50630)
0924 11:50:27.052352 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.052444 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 11:50:27.053653 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.053321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50640)
0924 11:50:27.053450 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.053528 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:50:27.053900 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:33087: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
W20260924 11:50:27.180428 15346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:27.180718 15346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:27.180845 15346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:27.188323 15346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:27.188426 15346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:27.188472 15346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:27.188556 15346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:27.188645 15346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:27.197172 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } attempt: 4
I20260924 11:50:27.198350 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.197842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50646)
0924 11:50:27.198124 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.198220 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 11:50:27.199296 15346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:35699,127.1.210.61:33087,127.1.210.60:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab
--principal=kudu/127.1.210.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:27.199375 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.199053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50654)
0924 11:50:27.199184 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.199259 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:50:27.199702 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:33087: Network error: Client connection negotiation failed: client connection to 127.1.210.61:33087: connect: Connection refused (error 111)
I20260924 11:50:27.200646 15346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:27.201984 15346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:27.213798 15351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:27.213791 15352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:27.214108 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:27.214700 15346 server_base.cc:1034] running on GCE node
I20260924 11:50:27.215479 15346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:27.216732 15346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:27.217932 15346 hybrid_clock.cc:648] HybridClock initialized: now 1790250627217894 us; error 33 us; skew 500 ppm
I20260924 11:50:27.218319 15346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:27.221504 15346 init.cc:377] Logged in from keytab as kudu/127.1.210.61@KRBTEST.COM (short username kudu)
I20260924 11:50:27.222855 15346 webserver.cc:460] Webserver started at http://127.1.210.61:44707/ using document root <none> and password file <none>
I20260924 11:50:27.223527 15346 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:27.223670 15346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:27.223984 15346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:27.226945 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data/instance:
uuid: "46e43d1ef7134df9b62991a1786683ac"
format_stamp: "Formatted at 2026-09-24 11:50:27 on dist-test-slave-dd32"
server_key: "859522fb57c286fa6e6a5e4a7bfe34ea"
server_key_iv: "6d273742cc4ac97181d092ead8137aed"
server_key_version: "encryptionkey@0"
I20260924 11:50:27.227731 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal/instance:
uuid: "46e43d1ef7134df9b62991a1786683ac"
format_stamp: "Formatted at 2026-09-24 11:50:27 on dist-test-slave-dd32"
server_key: "859522fb57c286fa6e6a5e4a7bfe34ea"
server_key_iv: "6d273742cc4ac97181d092ead8137aed"
server_key_version: "encryptionkey@0"
I20260924 11:50:27.232565 15346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:50:27.235808 15361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:27.237185 15346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:27.237368 15346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal
uuid: "46e43d1ef7134df9b62991a1786683ac"
format_stamp: "Formatted at 2026-09-24 11:50:27 on dist-test-slave-dd32"
server_key: "859522fb57c286fa6e6a5e4a7bfe34ea"
server_key_iv: "6d273742cc4ac97181d092ead8137aed"
server_key_version: "encryptionkey@0"
I20260924 11:50:27.237562 15346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:27.251137 15346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:27.255451 15346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:27.255722 15346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:27.274088 15346 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.61:33087
I20260924 11:50:27.274093 15412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.61:33087 every 8 connection(s)
I20260924 11:50:27.275478 15346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
I20260924 11:50:27.279162 15413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:27.284758 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15346
I20260924 11:50:27.284952 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-1/wal/instance
I20260924 11:50:27.285526 1864 external_mini_cluster.cc:1428] Setting key afbf08d17de8acd04440746051d41ec0
I20260924 11:50:27.285763 15413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.60@KRBTEST.COM; defaulting to no policy
I20260924 11:50:27.309661 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.288871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.1.210.62:35699)
0924 11:50:27.289105 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.289111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.290337 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 11:50:27.293173 (+ 2836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.293178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.293245 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.293332 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.295025 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.296213 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.298742 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.299173 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.299279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.303683 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.303688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.303691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.303723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.306459 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.307366 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.307369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.307372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.307515 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.308018 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.308021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.308022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.308294 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.308451 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.309198 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.309432 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:50:27.310664 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.289337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:35699 (local address 127.0.0.1:37918)
0924 11:50:27.290139 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.290188 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.292846 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.293585 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.293594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.294087 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.294829 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.294855 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.296441 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.296449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.298588 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.298605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.299404 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.299413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.299718 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.300454 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.300487 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.303457 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.306611 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.306621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.306640 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.307231 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.307628 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.307636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.307643 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.307895 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.308450 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.308459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.308817 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.309045 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.309523 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":305,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.60@KRBTEST.COM" created.
I20260924 11:50:27.312464 15413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:50:27.329007 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.313149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50662)
0924 11:50:27.313369 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.313390 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.313499 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.314521 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.314528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.315054 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.315264 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.315276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.316609 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.316617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.317848 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.317860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.318936 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.318945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.319210 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.319795 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.319822 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.322422 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.326679 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.326682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.326685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.327021 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.327436 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.327439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.327441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.327574 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.328335 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.328342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.328467 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.328568 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.328759 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 11:50:27.329635 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.313425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.1.210.61:33087)
0924 11:50:27.314097 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.314104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.314128 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:50:27.314172 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.314178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.314353 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.314466 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.315421 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.316462 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.317990 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.318783 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.318882 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.322654 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.322670 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.322691 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.322731 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.326535 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.327168 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.327180 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.327186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.327331 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.327695 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.327703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.327709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.328205 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.328426 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.328809 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.329125 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":397,"threads_started":1}
I20260924 11:50:27.331892 15413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:50:27.332937 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.332516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51722)
0924 11:50:27.332688 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.332805 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 11:50:27.333227 15362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.334769 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.334393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51726)
0924 11:50:27.334563 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.334674 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 11:50:27.334966 15413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab.
Entry for principal kudu/127.1.210.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab.
I20260924 11:50:27.340335 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:39091
--webserver_interface=127.1.210.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab
--principal=kudu/127.1.210.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:35699,127.1.210.61:33087,127.1.210.60:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:27.384318 15413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 1
I20260924 11:50:27.385725 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.385049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51734)
0924 11:50:27.385356 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.385506 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 11:50:27.386904 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.386428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51744)
0924 11:50:27.386618 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.386737 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:50:27.387189 15413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.455479 15413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 2
I20260924 11:50:27.456799 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.456223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51746)
0924 11:50:27.456472 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.456593 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 11:50:27.457849 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.457492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51750)
0924 11:50:27.457636 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.457714 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:50:27.458057 15413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.498956 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } attempt: 5
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:50:27.513584 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.499768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.1.210.61:33087)
0924 11:50:27.499954 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.499958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.499978 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:50:27.500176 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.500179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.500259 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.500354 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.501341 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.502031 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.503781 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.504183 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.504271 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.508423 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.508428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.508431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.508460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.511102 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.511851 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.511855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.511857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.511968 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.512341 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.512345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.512346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.512606 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.512714 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.513189 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.513369 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:50:27.513700 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.499653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50668)
0924 11:50:27.499893 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.499915 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.500046 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.500429 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.500434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.500936 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.501189 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.501203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.502185 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.502191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.503643 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.503659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.504348 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.504357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.504557 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.505250 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.505272 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.508213 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.511245 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.511251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.511255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.511728 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.512069 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.512074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.512078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.512239 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.512722 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.512727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.512904 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.513056 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.513356 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:50:27.514710 15339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:50:27.515848 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.515307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51764)
0924 11:50:27.515517 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.515632 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:50:27.516902 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.516514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51778)
0924 11:50:27.516688 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.516789 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 11:50:27.517144 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.549325 15413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 3
I20260924 11:50:27.550792 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.550312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51788)
0924 11:50:27.550528 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.550641 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 11:50:27.551870 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.551483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51800)
0924 11:50:27.551702 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.551769 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260924 11:50:27.552052 15413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.568388 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 1
I20260924 11:50:27.569597 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.568998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51816)
0924 11:50:27.569275 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.569387 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 11:50:27.570700 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.570264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51824)
0924 11:50:27.570435 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.570554 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:50:27.570958 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.639252 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 2
I20260924 11:50:27.640522 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.639898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51826)
0924 11:50:27.640150 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.640267 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 11:50:27.641741 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.641307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51836)
0924 11:50:27.641492 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.641584 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 11:50:27.641963 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
W20260924 11:50:27.672399 15420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:27.672673 15420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:27.672762 15420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:27.679643 15420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:27.679739 15420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:27.679785 15420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:27.679840 15420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:27.679890 15420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:27.689318 15420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:35699,127.1.210.61:33087,127.1.210.60:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab
--principal=kudu/127.1.210.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:27.690615 15420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:27.691834 15420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:50:27.695241 15413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 4
I20260924 11:50:27.696447 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.695911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51846)
0924 11:50:27.696218 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.696305 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 11:50:27.697541 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.697161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51858)
0924 11:50:27.697350 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.697429 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 11:50:27.697817 15413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
W20260924 11:50:27.704519 15428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:27.704605 15426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:27.704638 15420 server_base.cc:1034] running on GCE node
W20260924 11:50:27.704603 15425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:27.705842 15420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:27.707192 15420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:27.708431 15420 hybrid_clock.cc:648] HybridClock initialized: now 1790250627708390 us; error 54 us; skew 500 ppm
I20260924 11:50:27.708763 15420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:27.711764 15420 init.cc:377] Logged in from keytab as kudu/127.1.210.60@KRBTEST.COM (short username kudu)
I20260924 11:50:27.713205 15420 webserver.cc:460] Webserver started at http://127.1.210.60:33861/ using document root <none> and password file <none>
I20260924 11:50:27.713876 15420 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:27.714020 15420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:27.714323 15420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:27.717243 15420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data/instance:
uuid: "a21232a694e2483cbff5cb6979a76fd0"
format_stamp: "Formatted at 2026-09-24 11:50:27 on dist-test-slave-dd32"
server_key: "dda3d2a12d3c201da3958a9c877e019a"
server_key_iv: "b6fba11a43a8a618d700d15177fed7a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:27.718057 15420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal/instance:
uuid: "a21232a694e2483cbff5cb6979a76fd0"
format_stamp: "Formatted at 2026-09-24 11:50:27 on dist-test-slave-dd32"
server_key: "dda3d2a12d3c201da3958a9c877e019a"
server_key_iv: "b6fba11a43a8a618d700d15177fed7a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:27.723146 15420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:50:27.726430 15435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:27.727664 15420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:50:27.727840 15420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal
uuid: "a21232a694e2483cbff5cb6979a76fd0"
format_stamp: "Formatted at 2026-09-24 11:50:27 on dist-test-slave-dd32"
server_key: "dda3d2a12d3c201da3958a9c877e019a"
server_key_iv: "b6fba11a43a8a618d700d15177fed7a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:27.728040 15420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:27.748167 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 3
I20260924 11:50:27.749330 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.748818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51872)
0924 11:50:27.749029 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.749168 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:50:27.749863 15420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:27.750258 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.749925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51888)
0924 11:50:27.750097 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.750162 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:50:27.750802 15339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:39091: Network error: Client connection negotiation failed: client connection to 127.1.210.60:39091: connect: Connection refused (error 111)
I20260924 11:50:27.755036 15420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:27.755316 15420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:27.772940 15420 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.60:39091
I20260924 11:50:27.772972 15486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.60:39091 every 8 connection(s)
I20260924 11:50:27.774384 15420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
I20260924 11:50:27.778121 15487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:27.781455 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15420
I20260924 11:50:27.781654 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790250429340952-1864-0/minicluster-data/master-2/wal/instance
I20260924 11:50:27.782178 1864 external_mini_cluster.cc:1428] Setting key f789f88b07160a3789bfa0b6ad542bb0
I20260924 11:50:27.785830 15487 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:27.805198 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.789303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.1.210.62:35699)
0924 11:50:27.789508 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.789515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.789963 (+ 448us) server_negotiation.cc:407] Connection header received
0924 11:50:27.790140 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.790146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.790222 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.790337 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.791782 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.792689 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.794486 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.795224 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.795352 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.798843 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.798849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.798855 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.798891 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.802091 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.802858 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.802865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.802868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.803043 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.803517 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.803521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.803523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.803755 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.803920 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.804541 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.804893 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:50:27.806165 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.784043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.1.210.60:39091)
0924 11:50:27.784767 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.784783 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.784823 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:50:27.786481 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.786502 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.786898 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.787205 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.789014 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.790885 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.792418 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.793449 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.793606 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.798659 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.798669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.798685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.798724 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.801740 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.802540 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.802549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.802553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.802734 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.803156 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.803161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.803164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.803771 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.804073 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.804385 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.804778 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"spinlock_wait_cycles":19712,"thread_start_us":345,"threads_started":1}
I20260924 11:50:27.806187 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.789133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:35699 (local address 127.0.0.1:37930)
0924 11:50:27.789812 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.789850 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.790004 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.790398 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.790404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.791225 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.791599 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.791616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.792848 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.792855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.794265 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.794277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.795474 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.795482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.795693 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.796398 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.796431 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.798640 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.802213 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.802221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.802228 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.802695 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.803156 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.803161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.803164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.803402 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.803898 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.803905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.804110 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.804385 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.804902 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":130,"spinlock_wait_cycles":13568,"thread_start_us":305,"threads_started":1}
I20260924 11:50:27.808449 15487 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:27.823076 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.809109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:33087 (local address 127.0.0.1:50670)
0924 11:50:27.809445 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.809464 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.809555 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.809917 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.809925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.810294 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.810428 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.810435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.811679 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.811685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.812828 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.812839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.813773 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.813777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.813982 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.814521 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.814546 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.816738 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.820512 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.820516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.820519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.821016 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.821455 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.821461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.821465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.821611 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.822117 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.822122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.822268 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.822419 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.822697 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260924 11:50:27.823102 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.809348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.1.210.61:33087)
0924 11:50:27.809521 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.809527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.809548 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:50:27.809669 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.809674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.809757 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.809867 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.810578 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.811551 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.812967 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.813643 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.813773 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.816939 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.816946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.816953 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.816997 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.820378 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.821173 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.821180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.821184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.821341 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.821724 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.821730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.821733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.822000 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.822153 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.822510 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.822723 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:50:27.824174 15487 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:50:27.825172 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.809252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.1.210.62:35699)
0924 11:50:27.809428 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.809434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.809665 (+ 231us) server_negotiation.cc:407] Connection header received
0924 11:50:27.809804 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.809808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.809894 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.809989 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.811319 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.812333 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.813878 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.814508 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.814639 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.819022 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.819029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.819034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.819078 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.822044 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.822697 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.822702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.822707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.822840 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.823652 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.823657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.823660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.824030 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.824342 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.824545 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.824756 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:50:27.837908 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.824885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51896)
0924 11:50:27.825214 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.825353 (+ 139us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.825475 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.825747 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.825751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.826192 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.826341 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.826349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.827636 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.827640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.828533 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.828549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.829362 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.829371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.829564 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.830097 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.830112 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.832421 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.835438 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.835443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.835446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.835844 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.836224 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.836229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.836235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.836399 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.837012 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.837015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.837195 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.837345 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.837634 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260924 11:50:27.838223 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.825108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.1.210.60:39091)
0924 11:50:27.825257 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.825264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.825414 (+ 150us) server_negotiation.cc:407] Connection header received
0924 11:50:27.825583 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.825587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.825643 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.825733 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.826465 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.827506 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.828682 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.829222 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.829339 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.832609 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.832613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.832617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.832643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.835312 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.835954 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.835959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.835962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.836104 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.836523 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.836528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.836531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.836894 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.837043 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.837469 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.837835 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":56}
I20260924 11:50:27.841567 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.826778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.1.210.61:33087)
0924 11:50:27.826947 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.826951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.826966 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:50:27.827091 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.827094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.827147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.827220 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.828488 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.829328 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.830723 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.831390 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.831483 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.835634 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.835639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.835642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.835672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.838959 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.839616 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.839621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.839625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.839809 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.840251 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.840255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.840257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.840504 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.840747 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.841053 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.841284 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:50:27.844002 15487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: Bootstrap starting.
I20260924 11:50:27.846271 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:50:27.847357 15487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:27.848361 15487 log.cc:824] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:27.850942 15487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: No bootstrap required, opened a new log
I20260924 11:50:27.860739 15487 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:27.861228 15487 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:27.861370 15487 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21232a694e2483cbff5cb6979a76fd0, State: Initialized, Role: FOLLOWER
I20260924 11:50:27.862118 15487 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:27.866724 15500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:27.867231 15500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:27.868162 15487 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:27.871717 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.850931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.1.210.62:35699)
0924 11:50:27.851124 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.851130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.851621 (+ 491us) server_negotiation.cc:407] Connection header received
0924 11:50:27.851957 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.851966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.852070 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.852159 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.855020 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.856127 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.859044 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.860054 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.860182 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.864952 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.864963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.864970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.865023 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.868321 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.869170 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.869176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.869179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.869330 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.869832 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.869838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.869841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.870198 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.870456 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.871092 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.871341 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:50:27.875540 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.851821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.1.210.61:33087)
0924 11:50:27.851982 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.851987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.856809 (+ 4822us) server_negotiation.cc:407] Connection header received
0924 11:50:27.857102 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.857111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.857190 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.857260 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.859716 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.860749 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.863166 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.864088 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.864208 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.868323 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.868330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.868334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.868381 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.872434 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.873361 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.873366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.873371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.873517 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.874024 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.874029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.874031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.874392 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.874556 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.875009 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.875232 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:50:27.886157 15511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:50:27.886356 15511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:50:27.886989 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.856981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.1.210.60:39091)
0924 11:50:27.857220 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.857227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.868446 (+ 11219us) server_negotiation.cc:407] Connection header received
0924 11:50:27.868690 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.868699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.868787 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.868857 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.870928 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.872336 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.875079 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.876025 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.876143 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.879633 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.879652 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.879655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.879686 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.883368 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.884254 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.884259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.884262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.884459 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.884948 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.884958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.884963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.885437 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.885701 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.886305 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.886543 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":128}
I20260924 11:50:27.883291 15420 master_runner.cc:186] Error getting master registration for 127.1.210.62:35699: OK, instance_id { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" instance_seqno: 1790250626785464 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:50:27.893366 15420 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:50:27.928010 15339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 4
Sep 24 11:50:27 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250626, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:50:27.943269 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.928670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.1.210.60:39091)
0924 11:50:27.928855 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.928861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.928882 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:50:27.929014 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.929020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.929154 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.929259 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.929986 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.931235 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.932981 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.933530 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.933625 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.937513 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:27.937518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:27.937522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:27.937551 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:27.940467 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.941376 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.941380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.941382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.941520 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:27.941929 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:27.941933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:27.941935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:27.942213 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:27.942375 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:27.942845 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 11:50:27.943021 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:50:27.943457 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.928588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51902)
0924 11:50:27.928790 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.928808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.928896 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.929293 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.929296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.929664 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.929838 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.929847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.931402 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.931407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.932813 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.932828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:27.933685 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:27.933692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:27.933882 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:27.934534 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:27.934558 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:27.937301 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:27.940595 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.940602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.940630 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.941260 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.941637 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:27.941643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:27.941647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:27.941818 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:27.942363 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:27.942369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:27.942563 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 11:50:27.942726 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 11:50:27.943043 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:50:27.948539 15339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: Bootstrap starting.
I20260924 11:50:27.951941 15339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:27.952999 15339 log.cc:824] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:27.955722 15339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: No bootstrap required, opened a new log
I20260924 11:50:27.963465 15339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:27.963793 15339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:27.963923 15339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bb8b8851b34293ac35dd1e8d3a1032, State: Initialized, Role: FOLLOWER
I20260924 11:50:27.964447 15339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:27.965421 15514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:27.965783 15514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:27.966570 15339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:50:27.977285 15525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:50:27.977404 15525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:50:27.976199 15272 master_runner.cc:186] Error getting master registration for 127.1.210.61:33087: OK, instance_id { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" instance_seqno: 1790250627268729 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:50:27.987689 15272 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:50:27.997043 15413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } attempt: 5
I20260924 11:50:28.001833 15514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:28.002108 15514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:28.003744 15514 leader_election.cc:288] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46e43d1ef7134df9b62991a1786683ac (127.1.210.61:33087), a21232a694e2483cbff5cb6979a76fd0 (127.1.210.60:39091)
I20260924 11:50:28.005620 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a21232a694e2483cbff5cb6979a76fd0" is_pre_election: true
I20260924 11:50:28.006219 15462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87bb8b8851b34293ac35dd1e8d3a1032 in term 0.
W20260924 11:50:28.007300 15441 tablet.cc:2369] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 11:50:28 dist-test-slave-dd32 krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:50:28.010540 15288 leader_election.cc:302] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87bb8b8851b34293ac35dd1e8d3a1032, a21232a694e2483cbff5cb6979a76fd0; no voters:
I20260924 11:50:28.011557 15514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:50:28.011718 15514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:50:28.011822 15514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:28.016184 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.997691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:39091 (local address 127.0.0.1:51918)
0924 11:50:27.997979 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:27.998001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:27.998126 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:27.998542 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:27.998548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:27.999084 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:27.999314 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:27.999328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:28.000836 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:28.000842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:28.002318 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:28.002336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:28.004386 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:28.004392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:28.004615 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:28.005341 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:28.005364 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:28.008452 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:28.013142 (+ 4690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:28.013147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:28.013153 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:28.013747 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:28.014195 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:28.014199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:28.014203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:28.014387 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:28.015134 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:28.015139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:28.015313 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 11:50:28.015474 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:50:28.015774 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 11:50:28.016901 15514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:28.017705 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:27.997866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.1.210.60:39091)
0924 11:50:27.998059 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:27.998064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:27.998084 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:50:27.998301 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:27.998307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:27.998383 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:27.998485 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:27.999478 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:28.000675 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:28.002648 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:28.003371 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:28.003517 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:28.008677 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:28.008683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:28.008688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:28.008728 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:28.012954 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:28.013903 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:28.013909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:28.013913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:28.014055 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:28.014521 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:28.014526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:28.014530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:28.014914 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:28.015069 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:28.016663 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0924 11:50:28.016924 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:50:28.019059 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a21232a694e2483cbff5cb6979a76fd0"
I20260924 11:50:28.019300 15462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:28.022677 15462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87bb8b8851b34293ac35dd1e8d3a1032 in term 1.
I20260924 11:50:28.023131 15288 leader_election.cc:302] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87bb8b8851b34293ac35dd1e8d3a1032, a21232a694e2483cbff5cb6979a76fd0; no voters:
I20260924 11:50:28.024039 15413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: Bootstrap starting.
W20260924 11:50:28.024344 15293 tablet.cc:2369] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:50:28.028275 15413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:28.028926 15514 leader_election.cc:288] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [CANDIDATE]: Term 1 election: Requested vote from peers 46e43d1ef7134df9b62991a1786683ac (127.1.210.61:33087), a21232a694e2483cbff5cb6979a76fd0 (127.1.210.60:39091)
I20260924 11:50:28.029794 15514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:28.029938 15413 log.cc:824] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:28.030372 15514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [term 1 LEADER]: Becoming Leader. State: Replica: 87bb8b8851b34293ac35dd1e8d3a1032, State: Running, Role: LEADER
I20260924 11:50:28.032724 15514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:28.039654 15413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: No bootstrap required, opened a new log
I20260924 11:50:28.051132 15413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:28.051607 15413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:28.051756 15413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e43d1ef7134df9b62991a1786683ac, State: Initialized, Role: FOLLOWER
I20260924 11:50:28.052443 15413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } }
I20260924 11:50:28.053790 15529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.055124 15529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:28.056811 15387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e43d1ef7134df9b62991a1786683ac"
I20260924 11:50:28.057276 15387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:28.060632 15387 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87bb8b8851b34293ac35dd1e8d3a1032 in term 1.
I20260924 11:50:28.061103 15388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e43d1ef7134df9b62991a1786683ac" is_pre_election: true
I20260924 11:50:28.061440 15388 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 87bb8b8851b34293ac35dd1e8d3a1032 in term 1. Re-sending same reply.
I20260924 11:50:28.062393 15527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87bb8b8851b34293ac35dd1e8d3a1032. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.062737 15413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:28.062846 15527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:28.073776 15531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:28.076622 15531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:28.091439 15462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [term 1 FOLLOWER]: Refusing update from remote peer 87bb8b8851b34293ac35dd1e8d3a1032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:28.092667 15527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:28.102299 15387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [term 1 FOLLOWER]: Refusing update from remote peer 87bb8b8851b34293ac35dd1e8d3a1032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:28.103660 15527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:28.117825 15529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87bb8b8851b34293ac35dd1e8d3a1032. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.118235 15529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:28.120250 15500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87bb8b8851b34293ac35dd1e8d3a1032. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.120554 15500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:28.128780 15534 mvcc.cc:205] Tried to move back new op lower bound from 7332866572646883328 to 7332866572441804800. Current Snapshot: MvccSnapshot[applied={T|T < 7332866572646883328}]
I20260924 11:50:28.131735 15537 mvcc.cc:205] Tried to move back new op lower bound from 7332866572646883328 to 7332866572441804800. Current Snapshot: MvccSnapshot[applied={T|T < 7332866572646883328}]
I20260924 11:50:28.140224 15529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.140556 15529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:28.141218 15538 mvcc.cc:205] Tried to move back new op lower bound from 7332866572646883328 to 7332866572441804800. Current Snapshot: MvccSnapshot[applied={T|T < 7332866572646883328 or (T in {7332866572646883328})}]
I20260924 11:50:28.147424 15500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.147682 15500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:50:28.149048 15514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.149456 15514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:28.149806 15514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb8b8851b34293ac35dd1e8d3a1032" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 35699 } } peers { permanent_uuid: "46e43d1ef7134df9b62991a1786683ac" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 33087 } } peers { permanent_uuid: "a21232a694e2483cbff5cb6979a76fd0" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 39091 } } }
I20260924 11:50:28.150102 15514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:28.152340 15531 catalog_manager.cc:1347] Generated new cluster ID: 04e7220589a1455d981311ed23b9089c
I20260924 11:50:28.152462 15531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:28.203205 15531 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:50:28.205636 15531 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:28.213413 15550 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:50:28.222374 15531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87bb8b8851b34293ac35dd1e8d3a1032: Generated new TSK 0
I20260924 11:50:28.223651 15531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:50:28.888327 15511 catalog_manager.cc:1259] Loaded cluster ID: 04e7220589a1455d981311ed23b9089c
I20260924 11:50:28.888430 15511 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: loading cluster ID for follower catalog manager: success
I20260924 11:50:28.890779 15511 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: acquiring CA information for follower catalog manager: success
I20260924 11:50:28.892460 15511 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a21232a694e2483cbff5cb6979a76fd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:50:29.217741 15550 catalog_manager.cc:1259] Loaded cluster ID: 04e7220589a1455d981311ed23b9089c
I20260924 11:50:29.217866 15550 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: loading cluster ID for follower catalog manager: success
I20260924 11:50:29.220417 15550 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: acquiring CA information for follower catalog manager: success
I20260924 11:50:29.222471 15550 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46e43d1ef7134df9b62991a1786683ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:50:29.309631 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15272
I20260924 11:50:29.311070 15334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:29.423663 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15272
I20260924 11:50:29.444846 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15346
I20260924 11:50:29.446074 15408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:29.556569 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15346
I20260924 11:50:29.577345 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15420
I20260924 11:50:29.578521 15482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:29.685603 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15420
2026-09-24T11:50:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4649 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:50:29 dist-test-slave-dd32 krb5kdc[15562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:50:29 dist-test-slave-dd32 krb5kdc[15562](info): set up 2 sockets
Sep 24 11:50:29 dist-test-slave-dd32 krb5kdc[15562](info): commencing operation
krb5kdc: starting...
W20260924 11:50:30.821753 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:50:30 dist-test-slave-dd32 krb5kdc[15562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:50:31.056348 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:33679
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:31.379495 15575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:31.379787 15575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:31.379976 15575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:31.387228 15575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:31.387338 15575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:31.387451 15575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:31.387553 15575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:31.387634 15575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:31.397446 15575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:31.398779 15575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:31.400172 15575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:31.412271 15580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:31.412176 15581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:31.412227 15583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:31.413172 15575 server_base.cc:1034] running on GCE node
I20260924 11:50:31.413995 15575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:31.415387 15575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:31.416616 15575 hybrid_clock.cc:648] HybridClock initialized: now 1790250631416577 us; error 51 us; skew 500 ppm
I20260924 11:50:31.416962 15575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:31 dist-test-slave-dd32 krb5kdc[15562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:31.419947 15575 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:31.421707 15575 webserver.cc:460] Webserver started at http://127.1.210.62:45569/ using document root <none> and password file <none>
I20260924 11:50:31.422425 15575 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:31.422573 15575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:31.422909 15575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:31.426280 15575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "d4df913a0271442984a4e3db142dc675"
format_stamp: "Formatted at 2026-09-24 11:50:31 on dist-test-slave-dd32"
server_key: "5d68f62f91d71d95a0f0bb6d3c8518a9"
server_key_iv: "4d1ff62240269016f36a685eac4bb20e"
server_key_version: "encryptionkey@0"
I20260924 11:50:31.427065 15575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "d4df913a0271442984a4e3db142dc675"
format_stamp: "Formatted at 2026-09-24 11:50:31 on dist-test-slave-dd32"
server_key: "5d68f62f91d71d95a0f0bb6d3c8518a9"
server_key_iv: "4d1ff62240269016f36a685eac4bb20e"
server_key_version: "encryptionkey@0"
I20260924 11:50:31.432430 15575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:50:31.435890 15590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:31.437270 15575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:31.437458 15575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "d4df913a0271442984a4e3db142dc675"
format_stamp: "Formatted at 2026-09-24 11:50:31 on dist-test-slave-dd32"
server_key: "5d68f62f91d71d95a0f0bb6d3c8518a9"
server_key_iv: "4d1ff62240269016f36a685eac4bb20e"
server_key_version: "encryptionkey@0"
I20260924 11:50:31.437665 15575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:31.452986 15575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:31.456907 15575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:31.457253 15575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:31.475677 15575 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:33679
I20260924 11:50:31.475700 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:33679 every 8 connection(s)
I20260924 11:50:31.477226 15575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:31.477975 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15575
I20260924 11:50:31.478219 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:50:31.478992 1864 external_mini_cluster.cc:1428] Setting key 7742dc05bbfd37bf8ada914716af3283
I20260924 11:50:31.481771 15642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:31.492234 15642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675: Bootstrap starting.
Sep 24 11:50:31 dist-test-slave-dd32 krb5kdc[15562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:31.495961 15642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:31.497099 15642 log.cc:824] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:31.500356 15642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675: No bootstrap required, opened a new log
I20260924 11:50:31.501055 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:31.480838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.1.210.62:33679)
0924 11:50:31.481586 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:31.481598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:31.481640 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:50:31.483290 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:31.483311 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:31.483702 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:31.484105 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:31.485629 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:31.486931 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:31.488242 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:31.488908 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:31.489132 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:31.493673 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:31.493680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:31.493693 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:31.493730 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:31.496770 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:31.497710 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:31.497722 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:31.497735 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:31.497974 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:31.498394 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:31.498402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:31.498409 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:31.498929 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:31.499132 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:31.499603 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 11:50:31.499943 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
I20260924 11:50:31.505291 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:50:31.510361 15642 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4df913a0271442984a4e3db142dc675" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33679 } }
I20260924 11:50:31.510673 15642 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:31.510748 15642 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4df913a0271442984a4e3db142dc675, State: Initialized, Role: FOLLOWER
I20260924 11:50:31.511281 15642 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4df913a0271442984a4e3db142dc675" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33679 } }
I20260924 11:50:31.511487 15642 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:31.511596 15642 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:31.511714 15642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:31.513877 15642 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4df913a0271442984a4e3db142dc675" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33679 } }
I20260924 11:50:31.514294 15642 leader_election.cc:302] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4df913a0271442984a4e3db142dc675; no voters:
I20260924 11:50:31.514887 15642 leader_election.cc:288] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:50:31.515028 15647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:31.515702 15647 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [term 1 LEADER]: Becoming Leader. State: Replica: d4df913a0271442984a4e3db142dc675, State: Running, Role: LEADER
I20260924 11:50:31.516244 15647 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4df913a0271442984a4e3db142dc675" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33679 } }
I20260924 11:50:31.516909 15642 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:31.518947 15648 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4df913a0271442984a4e3db142dc675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4df913a0271442984a4e3db142dc675" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33679 } } }
I20260924 11:50:31.519063 15649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4df913a0271442984a4e3db142dc675. Latest consensus state: current_term: 1 leader_uuid: "d4df913a0271442984a4e3db142dc675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4df913a0271442984a4e3db142dc675" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 33679 } } }
I20260924 11:50:31.519428 15649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:31.519428 15648 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:31.521464 15654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:31.525940 15654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:31.533850 15654 catalog_manager.cc:1347] Generated new cluster ID: 686a858a29c347549eee3d34cac70a90
I20260924 11:50:31.533962 15654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:31.555228 15654 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:50:31.556569 15654 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:31.564961 15654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4df913a0271442984a4e3db142dc675: Generated new TSK 0
I20260924 11:50:31.565810 15654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:50:31.618286 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15575
I20260924 11:50:31.619421 15637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:31.715224 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15575
2026-09-24T11:50:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2025 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:50:31 dist-test-slave-dd32 krb5kdc[15670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:50:31 dist-test-slave-dd32 krb5kdc[15670](info): set up 2 sockets
Sep 24 11:50:31 dist-test-slave-dd32 krb5kdc[15670](info): commencing operation
krb5kdc: starting...
W20260924 11:50:32.817651 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:50:32 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:50:33.046689 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39227
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:33.371521 15683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:33.371806 15683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:33.371966 15683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:33.379276 15683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:33.379400 15683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:33.379495 15683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:33.379592 15683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:33.379669 15683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:33.389494 15683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:33.390799 15683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:33.392102 15683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:33.403857 15691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:33.403997 15688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:33.403975 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:33.404989 15683 server_base.cc:1034] running on GCE node
I20260924 11:50:33.405773 15683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:33.407094 15683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:33.408322 15683 hybrid_clock.cc:648] HybridClock initialized: now 1790250633408274 us; error 61 us; skew 500 ppm
I20260924 11:50:33.408686 15683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:33 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:33.411779 15683 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:33.413249 15683 webserver.cc:460] Webserver started at http://127.1.210.62:45551/ using document root <none> and password file <none>
I20260924 11:50:33.413863 15683 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:33.413959 15683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:33.414187 15683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:33.416934 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "329d886e40374bacaa6c39bd22c3c6a7"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "600e04715db2c3e8198c40877078b653"
server_key_iv: "ec5db06e2b60e8c1bbde3e710b2d3fc8"
server_key_version: "encryptionkey@0"
I20260924 11:50:33.417685 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "329d886e40374bacaa6c39bd22c3c6a7"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "600e04715db2c3e8198c40877078b653"
server_key_iv: "ec5db06e2b60e8c1bbde3e710b2d3fc8"
server_key_version: "encryptionkey@0"
I20260924 11:50:33.422478 15683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:50:33.425699 15698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:33.426915 15683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:33.427093 15683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "329d886e40374bacaa6c39bd22c3c6a7"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "600e04715db2c3e8198c40877078b653"
server_key_iv: "ec5db06e2b60e8c1bbde3e710b2d3fc8"
server_key_version: "encryptionkey@0"
I20260924 11:50:33.427264 15683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:33.440037 15683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:33.443710 15683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:33.443926 15683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:33.462464 15683 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:39227
I20260924 11:50:33.462484 15749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:39227 every 8 connection(s)
I20260924 11:50:33.463850 15683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:33.467532 15750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:33.468036 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15683
I20260924 11:50:33.468252 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:50:33.468899 1864 external_mini_cluster.cc:1428] Setting key 4a242e5b7798e9c233a66aad5a529c79
I20260924 11:50:33.481691 15750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap starting.
Sep 24 11:50:33 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:33.484957 15750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:33.486186 15750 log.cc:824] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:33.489748 15750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: No bootstrap required, opened a new log
I20260924 11:50:33.494082 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:33.470959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.1.210.62:39227)
0924 11:50:33.471817 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:33.471841 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:33.471902 (+ 61us) server_negotiation.cc:407] Connection header received
0924 11:50:33.474237 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:33.474265 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:33.474761 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:33.475212 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:33.476594 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:33.478632 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:33.480052 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:33.481052 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:33.481265 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:33.485622 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:33.485632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:33.485649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:33.485694 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:33.489554 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:33.490278 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:33.490286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:33.490296 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:33.490506 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:33.490863 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:33.490868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:33.490871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:33.491551 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:33.491890 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:33.492202 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 11:50:33.492705 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":365,"threads_started":1}
I20260924 11:50:33.499423 15750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:33.499737 15750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:33.499811 15750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329d886e40374bacaa6c39bd22c3c6a7, State: Initialized, Role: FOLLOWER
I20260924 11:50:33.500260 15750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:33.500456 15750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:33.500547 15750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:33.500666 15750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:33.502770 15750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:33.503191 15750 leader_election.cc:302] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 329d886e40374bacaa6c39bd22c3c6a7; no voters:
I20260924 11:50:33.503786 15750 leader_election.cc:288] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:50:33.503917 15755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:33.504750 15755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 LEADER]: Becoming Leader. State: Replica: 329d886e40374bacaa6c39bd22c3c6a7, State: Running, Role: LEADER
I20260924 11:50:33.505369 15755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:33.505996 15750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:33.508493 15757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329d886e40374bacaa6c39bd22c3c6a7. Latest consensus state: current_term: 1 leader_uuid: "329d886e40374bacaa6c39bd22c3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } } }
I20260924 11:50:33.508891 15757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:33.508617 15756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "329d886e40374bacaa6c39bd22c3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } } }
I20260924 11:50:33.509032 15756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:33.510988 15760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:33.516467 15760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:33.524196 15760 catalog_manager.cc:1347] Generated new cluster ID: a8e3df71bdd04f88bb00d3f218d61055
I20260924 11:50:33.524300 15760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:33.536392 15760 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:50:33.537704 15760 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:33.544653 15760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Generated new TSK 0
I20260924 11:50:33.545504 15760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:50:33.606863 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:39227
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:33.937206 15776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:33.937487 15776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:33.937676 15776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:33.944895 15776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:33.945041 15776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:33.945254 15776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:50:33.955742 15776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:33.957115 15776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:33.958454 15776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:33.971943 15782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:33.972002 15781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:33.972031 15784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:33.978797 15776 server_base.cc:1034] running on GCE node
I20260924 11:50:33.979441 15776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:33.980472 15776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:33.981762 15776 hybrid_clock.cc:648] HybridClock initialized: now 1790250633981697 us; error 62 us; skew 500 ppm
I20260924 11:50:33.982155 15776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:33 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:33.985958 15776 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:50:33.987540 15776 webserver.cc:460] Webserver started at http://127.1.210.1:42019/ using document root <none> and password file <none>
I20260924 11:50:33.988267 15776 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:33.988422 15776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:33.988729 15776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:33.991734 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "f9c8544e9b7c45409e38ae5d59f0f7de"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "07a07e6ab6c0fc89985934c71dd76353"
server_key_iv: "ae616d6bbc1642d92454a002d82199a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:33.992473 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "f9c8544e9b7c45409e38ae5d59f0f7de"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "07a07e6ab6c0fc89985934c71dd76353"
server_key_iv: "ae616d6bbc1642d92454a002d82199a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:33.997534 15776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 11:50:34.000974 15791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:34.002269 15776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:50:34.002442 15776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "f9c8544e9b7c45409e38ae5d59f0f7de"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "07a07e6ab6c0fc89985934c71dd76353"
server_key_iv: "ae616d6bbc1642d92454a002d82199a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:34.002645 15776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:34.021872 15776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:34.025797 15776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:34.026124 15776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:34.027066 15776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:34.028400 15776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:34.028505 15776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:34.028656 15776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:34.028729 15776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:34.062038 15776 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:40043
I20260924 11:50:34.062083 15903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:40043 every 8 connection(s)
I20260924 11:50:34.063337 15776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:50:34.071364 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15776
I20260924 11:50:34.071587 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:50:34.072247 1864 external_mini_cluster.cc:1428] Setting key 2d8a54409cead6a3b2731eed37fd4979
Sep 24 11:50:34 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:34.082818 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:34.066002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:53639 (local address 127.1.210.62:39227)
0924 11:50:34.066463 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:34.066468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:34.067333 (+ 865us) server_negotiation.cc:407] Connection header received
0924 11:50:34.069160 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:34.069166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:34.069252 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:34.069373 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:34.070821 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.071575 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.073220 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.073641 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.073778 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:34.077180 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:34.077186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:34.077189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:34.077220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:34.079767 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:34.080504 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:34.080512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:34.080517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:34.080647 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:34.081100 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:34.081106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:34.081109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:34.081384 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:34.081544 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:34.082207 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 11:50:34.082428 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":244,"threads_started":1}
I20260924 11:50:34.083798 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:34.066501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:53639)
0924 11:50:34.067178 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:34.067214 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:34.068869 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:34.069610 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:34.069618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:34.070114 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:34.070644 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:34.070660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.071734 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.071743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:34.073015 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:34.073027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.073873 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.073882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:34.074131 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:34.074809 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:34.074837 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:34.076946 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:34.079931 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:34.079939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:34.079954 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:34.080382 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:34.080761 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:34.080766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:34.080770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:34.080956 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:34.081513 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:34.081520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:34.081855 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 11:50:34.082120 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 11:50:34.082669 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260924 11:50:34.085368 15904 heartbeater.cc:347] Connected to a master server at 127.1.210.62:39227
I20260924 11:50:34.085686 15904 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:34.086586 15904 heartbeater.cc:511] Master 127.1.210.62:39227 requested a full tablet report, sending...
I20260924 11:50:34.088898 15715 ts_manager.cc:194] Registered new tserver with Master: f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1:40043)
I20260924 11:50:34.091112 15715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:53639
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:50:34.124763 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:39227
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:34.445748 15910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:34.446022 15910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:34.446115 15910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:34.453191 15910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:34.453297 15910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:34.453409 15910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:50:34.463017 15910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:34.464329 15910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:34.465695 15910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:34.479295 15916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:34.479385 15915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:34.479401 15918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:34.480126 15910 server_base.cc:1034] running on GCE node
I20260924 11:50:34.480705 15910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:34.481730 15910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:34.482950 15910 hybrid_clock.cc:648] HybridClock initialized: now 1790250634482906 us; error 42 us; skew 500 ppm
I20260924 11:50:34.483259 15910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:34 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:34.486393 15910 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:50:34.487746 15910 webserver.cc:460] Webserver started at http://127.1.210.2:33039/ using document root <none> and password file <none>
I20260924 11:50:34.488442 15910 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:34.488595 15910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:34.488905 15910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:34.491802 15910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "4ff777fc0add435faedc5d9ce3894768"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "99888fd41b7610403b90ec3c3aed6b97"
server_key_iv: "bf9970912f73d0f73c4943e65ae014dd"
server_key_version: "encryptionkey@0"
I20260924 11:50:34.492554 15910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "4ff777fc0add435faedc5d9ce3894768"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "99888fd41b7610403b90ec3c3aed6b97"
server_key_iv: "bf9970912f73d0f73c4943e65ae014dd"
server_key_version: "encryptionkey@0"
I20260924 11:50:34.497727 15910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:50:34.501178 15925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:34.502455 15910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:50:34.502614 15910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "4ff777fc0add435faedc5d9ce3894768"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "99888fd41b7610403b90ec3c3aed6b97"
server_key_iv: "bf9970912f73d0f73c4943e65ae014dd"
server_key_version: "encryptionkey@0"
I20260924 11:50:34.502808 15910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:34.525171 15910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:34.529733 15910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:34.530032 15910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:34.531054 15910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:34.532439 15910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:34.532514 15910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:34.532688 15910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:34.532739 15910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:34.564692 15910 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39417
I20260924 11:50:34.564738 16037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39417 every 8 connection(s)
I20260924 11:50:34.566037 15910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:50:34.576062 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 15910
I20260924 11:50:34.576254 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:50:34.576697 1864 external_mini_cluster.cc:1428] Setting key b3a2a5fe315c3a6a11bac61610c741bd
Sep 24 11:50:34 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:34.586951 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:34.568783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:51615 (local address 127.1.210.62:39227)
0924 11:50:34.568965 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:34.568969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:34.570213 (+ 1244us) server_negotiation.cc:407] Connection header received
0924 11:50:34.572116 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:34.572124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:34.572230 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:34.572345 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:34.573844 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.574547 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.576098 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.576610 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.576761 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:34.580854 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:34.580859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:34.580862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:34.580892 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:34.583429 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:34.584360 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:34.584363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:34.584366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:34.584486 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:34.584974 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:34.584977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:34.584980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:34.585286 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:34.585397 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:34.586257 (+ 860us) server_negotiation.cc:299] Negotiation successful
0924 11:50:34.586530 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:50:34.587774 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:34.569362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:51615)
0924 11:50:34.570055 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:34.570092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:34.571726 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:34.572572 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:34.572581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:34.573127 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:34.573683 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:34.573703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.574698 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.574703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:34.575951 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:34.575961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:34.576842 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:34.576848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:34.577127 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:34.577804 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:34.577833 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:34.580593 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:34.583587 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:34.583598 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:34.583618 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:34.584242 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:34.584592 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:34.584600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:34.584606 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:34.584849 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:34.585411 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:34.585422 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:34.585875 (+ 453us) client_negotiation.cc:769] Sending connection context
0924 11:50:34.586182 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 11:50:34.586646 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":270,"threads_started":1}
I20260924 11:50:34.589305 16038 heartbeater.cc:347] Connected to a master server at 127.1.210.62:39227
I20260924 11:50:34.589623 16038 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:34.590474 16038 heartbeater.cc:511] Master 127.1.210.62:39227 requested a full tablet report, sending...
I20260924 11:50:34.592036 15715 ts_manager.cc:194] Registered new tserver with Master: 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417)
I20260924 11:50:34.592813 15715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:51615
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:50:34.631914 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:39227
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:34.957033 16043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:34.957376 16043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:34.957471 16043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:34.964561 16043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:34.964659 16043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:34.964777 16043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:50:34.974710 16043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:34.975929 16043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:34.977293 16043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:34.990139 16048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:34.990108 16051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:34.990098 16049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:34.991091 16043 server_base.cc:1034] running on GCE node
I20260924 11:50:34.991751 16043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:34.992714 16043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:34.993935 16043 hybrid_clock.cc:648] HybridClock initialized: now 1790250634993898 us; error 50 us; skew 500 ppm
I20260924 11:50:34.994251 16043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:34 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:34.997581 16043 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:50:34.998862 16043 webserver.cc:460] Webserver started at http://127.1.210.3:45199/ using document root <none> and password file <none>
I20260924 11:50:34.999491 16043 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:34.999586 16043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:34.999819 16043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:35.002934 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "0a05841940da42c0922ae62c0d6fc9d6"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "d3784dca7c18fc68a6a9edff7d3e6257"
server_key_iv: "13bbef77c0103adfd5db5c31ebe54eb9"
server_key_version: "encryptionkey@0"
I20260924 11:50:35.003648 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "0a05841940da42c0922ae62c0d6fc9d6"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "d3784dca7c18fc68a6a9edff7d3e6257"
server_key_iv: "13bbef77c0103adfd5db5c31ebe54eb9"
server_key_version: "encryptionkey@0"
I20260924 11:50:35.009016 16043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 11:50:35.012683 16058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:35.014048 16043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:35.014171 16043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "0a05841940da42c0922ae62c0d6fc9d6"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "d3784dca7c18fc68a6a9edff7d3e6257"
server_key_iv: "13bbef77c0103adfd5db5c31ebe54eb9"
server_key_version: "encryptionkey@0"
I20260924 11:50:35.014335 16043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:35.039613 16043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:35.043965 16043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:35.044210 16043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:35.045532 16043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:35.046949 16043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:35.047021 16043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:35.047101 16043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:35.047142 16043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:35.079850 16043 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:43857
I20260924 11:50:35.079931 16170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:43857 every 8 connection(s)
I20260924 11:50:35.081210 16043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:50:35.083663 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16043
I20260924 11:50:35.083859 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:50:35.084462 1864 external_mini_cluster.cc:1428] Setting key f95267e05632d6428c83c7d55714487d
I20260924 11:50:35.094834 15904 heartbeater.cc:503] Master 127.1.210.62:39227 was elected leader, sending a full tablet report...
Sep 24 11:50:35 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:35.101619 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.083871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:47077 (local address 127.1.210.62:39227)
0924 11:50:35.084080 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.084084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.085452 (+ 1368us) server_negotiation.cc:407] Connection header received
0924 11:50:35.087904 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.087909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.088028 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.088136 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:35.089652 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.090410 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.091909 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.092301 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.092444 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.095850 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:35.095857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:35.095861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:35.095892 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:35.098762 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.099656 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.099661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.099664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.099823 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.100228 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.100231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.100233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.100484 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:35.100590 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.101225 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.101375 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 11:50:35.102555 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.084400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:47077)
0924 11:50:35.085280 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:35.085329 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:35.087632 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:35.088363 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:35.088371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:35.088853 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:35.089489 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.089506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.090559 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.090564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.091750 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.091765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.092526 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.092534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.092771 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.093405 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:35.093427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:35.095656 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:35.098904 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:35.098912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:35.098926 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:35.099514 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:35.099957 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:35.099961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:35.099963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:35.100129 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:35.100590 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:35.100595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:35.100859 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 11:50:35.101121 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 11:50:35.101503 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":355,"threads_started":1}
I20260924 11:50:35.103787 16171 heartbeater.cc:347] Connected to a master server at 127.1.210.62:39227
I20260924 11:50:35.104094 16171 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:35.104766 16171 heartbeater.cc:511] Master 127.1.210.62:39227 requested a full tablet report, sending...
I20260924 11:50:35.106274 15715 ts_manager.cc:194] Registered new tserver with Master: 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
I20260924 11:50:35.107263 15715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:47077
I20260924 11:50:35.111809 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:50:35.128059 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.116295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.1.210.62:39227)
0924 11:50:35.116475 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.116480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.116922 (+ 442us) server_negotiation.cc:407] Connection header received
0924 11:50:35.117155 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.117160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.117230 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.117309 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:35.118741 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.119441 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.120670 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.121052 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.121205 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.123359 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:35.123364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:35.123367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:35.123394 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:35.125988 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.126584 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.126587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.126590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.126703 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.127000 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.127003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.127005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.127286 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:35.127476 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.127679 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.127829 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:50:35.131825 15715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:50:35.138438 15715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:50:35.175644 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.151336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.1.210.1:40043)
0924 11:50:35.164203 (+ 12867us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.164216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.166069 (+ 1853us) server_negotiation.cc:407] Connection header received
0924 11:50:35.166164 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.166170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.166363 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.166523 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:35.168524 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.170023 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.172861 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.174434 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.174586 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.174824 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.175113 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.175297 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12614,"thread_start_us":12874,"threads_started":1}
I20260924 11:50:35.175693 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.150652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35598)
0924 11:50:35.165800 (+ 15148us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:35.165817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:35.165944 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:35.167831 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:35.167851 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:35.167878 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:35.168302 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.168314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.170182 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.170187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.172666 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.172686 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.174641 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.174647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.174839 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.174900 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 11:50:35.175045 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 11:50:35.175228 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15034,"mutex_wait_us":4,"thread_start_us":13967,"threads_started":1}
I20260924 11:50:35.175889 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.150358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.1.210.3:43857)
0924 11:50:35.165800 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.165809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.165848 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:50:35.165952 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.165959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.166231 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.166438 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:35.167494 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.169600 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.173154 (+ 3554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.174503 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.174610 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.174823 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.175363 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.175532 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15269,"thread_start_us":15098,"threads_started":1}
I20260924 11:50:35.176702 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.150239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43857 (local address 127.0.0.1:34448)
0924 11:50:35.165010 (+ 14771us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:35.165046 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:35.165255 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:35.166781 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:35.166788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:35.166823 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:35.167278 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.167291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.169763 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.169768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.172959 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.172972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.174965 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.174970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.175160 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.175191 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:50:35.176129 (+ 938us) client_negotiation.cc:240] Negotiation successful
0924 11:50:35.176268 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14514,"mutex_wait_us":699,"thread_start_us":15311,"threads_started":1}
I20260924 11:50:35.180271 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.165974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39417 (local address 127.0.0.1:60462)
0924 11:50:35.172237 (+ 6263us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:35.172255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:35.172381 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:35.173548 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:35.173552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:35.173578 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:35.173928 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.173936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.175581 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.175586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.177512 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.177524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.179575 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.179581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.179738 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.179769 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:50:35.179856 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:50:35.179984 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6132,"thread_start_us":5996,"threads_started":1}
I20260924 11:50:35.179770 15839 tablet_service.cc:1467] Processing CreateTablet for tablet f3d40343b240474abf075d7701f11d8a (DEFAULT_TABLE table=test-table [id=24648a09b4fe402b975e1ae44dca084d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:35.180720 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.166252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.1.210.2:39417)
0924 11:50:35.172594 (+ 6342us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.172600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.172623 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:50:35.172758 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.172768 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.173037 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.173260 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:35.174096 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.175417 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.177665 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.179107 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.179230 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.179441 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.180111 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.180275 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6182,"thread_start_us":6069,"threads_started":1}
I20260924 11:50:35.181461 15839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d40343b240474abf075d7701f11d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:35.180500 16106 tablet_service.cc:1467] Processing CreateTablet for tablet f3d40343b240474abf075d7701f11d8a (DEFAULT_TABLE table=test-table [id=24648a09b4fe402b975e1ae44dca084d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:35.182492 16106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d40343b240474abf075d7701f11d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:35.184396 15973 tablet_service.cc:1467] Processing CreateTablet for tablet f3d40343b240474abf075d7701f11d8a (DEFAULT_TABLE table=test-table [id=24648a09b4fe402b975e1ae44dca084d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:35.185658 15973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d40343b240474abf075d7701f11d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:35.192832 16189 tablet_bootstrap.cc:492] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Bootstrap starting.
I20260924 11:50:35.195880 16189 tablet_bootstrap.cc:654] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:35.197150 16190 tablet_bootstrap.cc:492] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Bootstrap starting.
I20260924 11:50:35.197122 16189 log.cc:824] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:35.200482 16190 tablet_bootstrap.cc:654] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:35.200769 16192 tablet_bootstrap.cc:492] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Bootstrap starting.
I20260924 11:50:35.201867 16190 log.cc:824] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:35.202059 16189 tablet_bootstrap.cc:492] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: No bootstrap required, opened a new log
I20260924 11:50:35.202322 16189 ts_tablet_manager.cc:1397] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:50:35.204066 16192 tablet_bootstrap.cc:654] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:35.204818 16190 tablet_bootstrap.cc:492] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: No bootstrap required, opened a new log
I20260924 11:50:35.205005 16192 log.cc:824] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:35.205085 16190 ts_tablet_manager.cc:1397] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 11:50:35.207670 16192 tablet_bootstrap.cc:492] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: No bootstrap required, opened a new log
I20260924 11:50:35.207949 16192 ts_tablet_manager.cc:1397] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 11:50:35.212877 16189 raft_consensus.cc:348] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.213588 16189 raft_consensus.cc:374] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:35.213747 16189 raft_consensus.cc:729] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c8544e9b7c45409e38ae5d59f0f7de, State: Initialized, Role: FOLLOWER
I20260924 11:50:35.215008 16189 consensus_queue.cc:260] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.215638 16192 raft_consensus.cc:348] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.215505 16190 raft_consensus.cc:348] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.215981 16192 raft_consensus.cc:374] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:35.216061 16190 raft_consensus.cc:374] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:35.216073 16192 raft_consensus.cc:729] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff777fc0add435faedc5d9ce3894768, State: Initialized, Role: FOLLOWER
I20260924 11:50:35.216167 16190 raft_consensus.cc:729] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a05841940da42c0922ae62c0d6fc9d6, State: Initialized, Role: FOLLOWER
I20260924 11:50:35.216691 16192 consensus_queue.cc:260] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.216945 16190 consensus_queue.cc:260] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.218367 16171 heartbeater.cc:503] Master 127.1.210.62:39227 was elected leader, sending a full tablet report...
I20260924 11:50:35.219262 16190 ts_tablet_manager.cc:1428] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260924 11:50:35.220796 16189 ts_tablet_manager.cc:1428] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260924 11:50:35.221659 16038 heartbeater.cc:503] Master 127.1.210.62:39227 was elected leader, sending a full tablet report...
I20260924 11:50:35.222261 16192 ts_tablet_manager.cc:1428] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
W20260924 11:50:35.315848 15905 tablet.cc:2369] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:50:35.318288 16039 tablet.cc:2369] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:50:35.332752 16172 tablet.cc:2369] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:50:35.370821 16196 raft_consensus.cc:482] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:35.371094 16196 raft_consensus.cc:504] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.372792 16196 leader_election.cc:288] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857), f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1:40043)
I20260924 11:50:35.379627 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.373255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57371 (local address 127.1.210.3:43857)
0924 11:50:35.373511 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.373521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.374398 (+ 877us) server_negotiation.cc:407] Connection header received
0924 11:50:35.374595 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.374599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.374652 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.374747 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:35.375341 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.376054 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.377847 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.378575 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.378730 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.378863 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.379229 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.379362 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:50:35.379630 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.373518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:42413 (local address 127.1.210.1:40043)
0924 11:50:35.373766 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.373774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.374174 (+ 400us) server_negotiation.cc:407] Connection header received
0924 11:50:35.374419 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.374423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.374482 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.374585 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:35.375217 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.375959 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.377799 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.378580 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.378742 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.378859 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.379228 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.379372 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:50:35.379840 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.373139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.1.210.2:42413)
0924 11:50:35.374053 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:35.374071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:35.374251 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:35.374603 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:35.374607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:35.374639 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:35.375059 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.375070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.376090 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.376098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.377649 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.377661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.378863 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.378871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.379046 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.379096 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 11:50:35.379204 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:50:35.379372 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":48,"spinlock_wait_cycles":28928,"thread_start_us":851,"threads_started":1}
I20260924 11:50:35.380199 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.373214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43857 (local address 127.1.210.2:57371)
0924 11:50:35.374277 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:35.374295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:35.374460 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:35.374762 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:35.374766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:35.374804 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:35.375194 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.375202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.376196 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.376201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.377687 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:35.377699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.378816 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.378823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:35.378990 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.379098 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 11:50:35.379195 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:50:35.379371 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":80,"thread_start_us":511,"threads_started":1}
I20260924 11:50:35.380899 15859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d40343b240474abf075d7701f11d8a" candidate_uuid: "4ff777fc0add435faedc5d9ce3894768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" is_pre_election: true
I20260924 11:50:35.381242 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d40343b240474abf075d7701f11d8a" candidate_uuid: "4ff777fc0add435faedc5d9ce3894768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a05841940da42c0922ae62c0d6fc9d6" is_pre_election: true
I20260924 11:50:35.381531 15859 raft_consensus.cc:2454] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff777fc0add435faedc5d9ce3894768 in term 0.
I20260924 11:50:35.381754 16126 raft_consensus.cc:2454] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff777fc0add435faedc5d9ce3894768 in term 0.
I20260924 11:50:35.382241 15929 leader_election.cc:302] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff777fc0add435faedc5d9ce3894768, f9c8544e9b7c45409e38ae5d59f0f7de; no voters:
I20260924 11:50:35.382612 16196 raft_consensus.cc:2781] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:50:35.382757 16196 raft_consensus.cc:482] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:50:35.382879 16196 raft_consensus.cc:3037] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:35.385253 16196 raft_consensus.cc:504] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.385799 16196 leader_election.cc:288] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [CANDIDATE]: Term 1 election: Requested vote from peers 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857), f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1:40043)
I20260924 11:50:35.386152 16126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d40343b240474abf075d7701f11d8a" candidate_uuid: "4ff777fc0add435faedc5d9ce3894768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a05841940da42c0922ae62c0d6fc9d6"
I20260924 11:50:35.386246 15859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d40343b240474abf075d7701f11d8a" candidate_uuid: "4ff777fc0add435faedc5d9ce3894768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de"
I20260924 11:50:35.386334 16126 raft_consensus.cc:3037] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:35.386467 15859 raft_consensus.cc:3037] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:35.388569 16126 raft_consensus.cc:2454] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff777fc0add435faedc5d9ce3894768 in term 1.
I20260924 11:50:35.388928 15927 leader_election.cc:302] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a05841940da42c0922ae62c0d6fc9d6, 4ff777fc0add435faedc5d9ce3894768; no voters:
I20260924 11:50:35.389156 15859 raft_consensus.cc:2454] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff777fc0add435faedc5d9ce3894768 in term 1.
I20260924 11:50:35.389215 16196 raft_consensus.cc:2781] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:35.389793 16196 raft_consensus.cc:686] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff777fc0add435faedc5d9ce3894768, State: Running, Role: LEADER
I20260924 11:50:35.390336 16196 consensus_queue.cc:237] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:35.394603 15715 catalog_manager.cc:5495] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "4ff777fc0add435faedc5d9ce3894768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } health_report { overall_health: UNKNOWN } } }
I20260924 11:50:35.456969 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.451454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.1.210.2:39417)
0924 11:50:35.451673 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.451678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.451696 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:50:35.451785 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.451789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.451876 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.452002 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:50:35.452417 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.453213 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.454693 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.455127 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.455221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.455759 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:50:35.456309 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:50:35.456438 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.456597 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.456673 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:50:35.465312 16126 raft_consensus.cc:1261] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Refusing update from remote peer 4ff777fc0add435faedc5d9ce3894768: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:35.465381 15859 raft_consensus.cc:1261] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Refusing update from remote peer 4ff777fc0add435faedc5d9ce3894768: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:35.466310 16200 consensus_queue.cc:1035] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:35.466671 16196 consensus_queue.cc:1035] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:35.496927 15715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:50:35.497288 15715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:50:35.502602 15715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Sending DeleteTablet for 3 replicas of tablet f3d40343b240474abf075d7701f11d8a
I20260924 11:50:35.503962 16106 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d40343b240474abf075d7701f11d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:35 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:34448
I20260924 11:50:35.504554 15973 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d40343b240474abf075d7701f11d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:35 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:60462
I20260924 11:50:35.507009 15839 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d40343b240474abf075d7701f11d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:35 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:35598
I20260924 11:50:35.510965 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15776
I20260924 11:50:35.542330 16215 tablet_replica.cc:330] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: stopping tablet replica
I20260924 11:50:35.546068 16214 tablet_replica.cc:330] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: stopping tablet replica
I20260924 11:50:35.546654 16214 raft_consensus.cc:2229] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:50:35.549373 16213 tablet_replica.cc:330] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: stopping tablet replica
I20260924 11:50:35.549880 16213 raft_consensus.cc:2229] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:50:35.554404 16213 raft_consensus.cc:2258] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:35.557298 16213 ts_tablet_manager.cc:1905] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:35.578920 16214 raft_consensus.cc:2258] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:35.579185 16215 raft_consensus.cc:2229] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:50:35.581609 16214 ts_tablet_manager.cc:1905] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:35.582850 16215 raft_consensus.cc:2258] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:35.585114 16215 ts_tablet_manager.cc:1905] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:50:35 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:50:35.591938 16215 ts_tablet_manager.cc:1918] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:35.592101 16215 log.cc:1196] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/f3d40343b240474abf075d7701f11d8a
I20260924 11:50:35.592669 16215 ts_tablet_manager.cc:1939] T f3d40343b240474abf075d7701f11d8a P f9c8544e9b7c45409e38ae5d59f0f7de: Deleting consensus metadata
I20260924 11:50:35.598853 16214 ts_tablet_manager.cc:1918] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:35.599030 16214 log.cc:1196] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/f3d40343b240474abf075d7701f11d8a
I20260924 11:50:35.599579 16214 ts_tablet_manager.cc:1939] T f3d40343b240474abf075d7701f11d8a P 4ff777fc0add435faedc5d9ce3894768: Deleting consensus metadata
I20260924 11:50:35.600488 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.512125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.1.210.1:40043)
0924 11:50:35.512329 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.512352 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.543740 (+ 31388us) server_negotiation.cc:407] Connection header received
0924 11:50:35.544096 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.544105 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.544225 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.544321 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:35.550739 (+ 6418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.552308 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.562482 (+ 10174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.563571 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.563728 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.592310 (+ 28582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:35.592324 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:35.592341 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:35.592382 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:35.596333 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.597288 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.597297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.597306 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.597473 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.597966 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.597971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.597975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.598396 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:35.598656 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.599902 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.600132 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:50:35.601464 15700 catalog_manager.cc:4841] TS 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): tablet f3d40343b240474abf075d7701f11d8a (table test-table [id=24648a09b4fe402b975e1ae44dca084d]) successfully deleted
I20260924 11:50:35.602026 15899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:35.604236 16213 ts_tablet_manager.cc:1918] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:35.604519 16213 log.cc:1196] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/f3d40343b240474abf075d7701f11d8a
I20260924 11:50:35.605140 16213 ts_tablet_manager.cc:1939] T f3d40343b240474abf075d7701f11d8a P 0a05841940da42c0922ae62c0d6fc9d6: Deleting consensus metadata
I20260924 11:50:35.607014 15700 catalog_manager.cc:4841] TS 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): tablet f3d40343b240474abf075d7701f11d8a (table test-table [id=24648a09b4fe402b975e1ae44dca084d]) successfully deleted
I20260924 11:50:35.707564 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15776
I20260924 11:50:35.707643 15702 catalog_manager.cc:4841] TS f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1:40043): tablet f3d40343b240474abf075d7701f11d8a (table test-table [id=24648a09b4fe402b975e1ae44dca084d]) successfully deleted
I20260924 11:50:35.731899 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15910
Sep 24 11:50:35 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:50:35.748890 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.732711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.1.210.2:39417)
0924 11:50:35.732908 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.732914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.732936 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:50:35.733231 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.733237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.733318 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.733419 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:35.734673 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.735885 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.737223 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.737899 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.738032 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.742255 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:35.742271 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:35.742290 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:35.742340 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:35.746169 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.746813 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.746824 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.746833 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.746997 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.747319 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.747325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.747328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.747774 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:35.748110 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.748248 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.748463 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:50:35.749641 16033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:35.864940 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15910
I20260924 11:50:35.883216 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16043
Sep 24 11:50:35 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:50:35.898432 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:35.883980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.1.210.3:43857)
0924 11:50:35.884449 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:35.884454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:35.884468 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:50:35.884520 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:35.884524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:35.884588 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:35.884701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:35.886065 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.886797 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.888086 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:35.888483 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:35.888614 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:35.892913 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:35.892926 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:35.892943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:35.892973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:35.895446 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.896142 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.896151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.896153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.896279 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:35.896653 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:35.896659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:35.896662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:35.896999 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:35.897549 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:35.897830 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 11:50:35.898065 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260924 11:50:35.899077 16166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:35.989748 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16043
I20260924 11:50:36.006331 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 15683
I20260924 11:50:36.007524 15745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:36.117904 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 15683
I20260924 11:50:36.139071 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39227
--webserver_interface=127.1.210.62
--webserver_port=45551
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:36.465391 16222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:36.465684 16222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:36.465772 16222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:36.472996 16222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:36.473114 16222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:36.473196 16222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:36.473276 16222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:36.473321 16222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:36.483213 16222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=45551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:36.484459 16222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:36.485711 16222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:36.509904 16231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:36.509935 16229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:36.509989 16228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:36.511049 16222 server_base.cc:1034] running on GCE node
I20260924 11:50:36.512037 16222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:36.513823 16222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:36.515187 16222 hybrid_clock.cc:648] HybridClock initialized: now 1790250636515128 us; error 28 us; skew 500 ppm
I20260924 11:50:36.515620 16222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:36 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:36.519258 16222 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:36.520891 16222 webserver.cc:460] Webserver started at http://127.1.210.62:45551/ using document root <none> and password file <none>
I20260924 11:50:36.521706 16222 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:36.521873 16222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:36.528579 16222 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 11:50:36.531713 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:36.533201 16222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:50:36.533383 16222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "329d886e40374bacaa6c39bd22c3c6a7"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "600e04715db2c3e8198c40877078b653"
server_key_iv: "ec5db06e2b60e8c1bbde3e710b2d3fc8"
server_key_version: "encryptionkey@0"
I20260924 11:50:36.534533 16222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:36.558502 16222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:36.562800 16222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:36.563092 16222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:36.581964 16222 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:39227
I20260924 11:50:36.581976 16289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:39227 every 8 connection(s)
I20260924 11:50:36.583525 16222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:36.590387 16290 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:50:36.591689 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16222
I20260924 11:50:36.592463 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:40043
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:39227
--webserver_port=42019
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:36.593734 16290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap starting.
I20260924 11:50:36.617865 16290 log.cc:824] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:36.629948 16290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:50:36.630477 16290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap complete.
I20260924 11:50:36.639719 16290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:36.640532 16290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329d886e40374bacaa6c39bd22c3c6a7, State: Initialized, Role: FOLLOWER
I20260924 11:50:36.641166 16290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:36.641417 16290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:36.641547 16290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:36.641724 16290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:50:36.645462 16290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:36.645994 16290 leader_election.cc:302] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 329d886e40374bacaa6c39bd22c3c6a7; no voters:
I20260924 11:50:36.646637 16290 leader_election.cc:288] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:50:36.646773 16294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:50:36.648047 16294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 LEADER]: Becoming Leader. State: Replica: 329d886e40374bacaa6c39bd22c3c6a7, State: Running, Role: LEADER
I20260924 11:50:36.648598 16294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:36.648978 16290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:36.651198 16296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329d886e40374bacaa6c39bd22c3c6a7. Latest consensus state: current_term: 2 leader_uuid: "329d886e40374bacaa6c39bd22c3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } } }
I20260924 11:50:36.651232 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "329d886e40374bacaa6c39bd22c3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } } }
I20260924 11:50:36.651580 16296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:36.651667 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:36.652966 16301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:36.671833 16301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:36.673655 16301 catalog_manager.cc:1259] Loaded cluster ID: a8e3df71bdd04f88bb00d3f218d61055
I20260924 11:50:36.673947 16301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:36.677269 16301 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 11:50:36.695182 16301 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 11:50:36.712069 16301 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:36.713949 16301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Loaded TSK: 0
I20260924 11:50:36.715536 16301 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:50:36.978987 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:36.979306 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:36.979480 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:36.987739 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:36.987890 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:36.988071 16292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:50:36.997885 16292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:40043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=42019
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:36.999233 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:37.000581 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:37.017719 16317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:37.021485 16318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:37.021533 16320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:37.022866 16292 server_base.cc:1034] running on GCE node
I20260924 11:50:37.023572 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:37.024626 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:37.025844 16292 hybrid_clock.cc:648] HybridClock initialized: now 1790250637025782 us; error 52 us; skew 500 ppm
I20260924 11:50:37.026191 16292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:37 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:37.034622 16292 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:50:37.036406 16292 webserver.cc:460] Webserver started at http://127.1.210.1:42019/ using document root <none> and password file <none>
I20260924 11:50:37.037254 16292 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:37.037408 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:37.043154 16292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:50:37.045922 16327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:37.047309 16292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:37.047474 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "f9c8544e9b7c45409e38ae5d59f0f7de"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "07a07e6ab6c0fc89985934c71dd76353"
server_key_iv: "ae616d6bbc1642d92454a002d82199a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:37.048437 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:37.063228 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:37.067619 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:37.067900 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:37.068949 16292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:37.070436 16292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:37.070549 16292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:37.070672 16292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:37.070765 16292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:37.112932 16292 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:40043
I20260924 11:50:37.113145 16439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:40043 every 8 connection(s)
I20260924 11:50:37.114584 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:50:37.120841 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16292
I20260924 11:50:37.121795 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:39417
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:39227
--webserver_port=33039
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:50:37 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:37.167129 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:37.118406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:50519 (local address 127.1.210.62:39227)
0924 11:50:37.138546 (+ 20140us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:37.138566 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:37.139829 (+ 1263us) server_negotiation.cc:407] Connection header received
0924 11:50:37.142740 (+ 2911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:37.142761 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:37.143273 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:37.143661 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:37.146209 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.148417 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.151370 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.152648 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.152794 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:37.157739 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:37.157750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:37.157775 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:37.157818 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:37.161962 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:37.162894 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:37.162903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:37.162911 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:37.163090 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:37.163618 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:37.163623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:37.163626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:37.164285 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:37.164456 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:37.165464 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0924 11:50:37.165913 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19860,"thread_start_us":21419,"threads_started":1}
I20260924 11:50:37.169909 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:37.118897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:50519)
0924 11:50:37.139265 (+ 20368us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:37.139334 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:37.142074 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:37.144484 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:37.144494 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:37.145220 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:37.145950 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:37.145973 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.148653 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.148658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:37.151190 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:37.151208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.153216 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.153223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:37.153494 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:37.154343 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:37.154373 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:37.157427 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:37.162140 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:37.162148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:37.162161 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:37.162739 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:37.163236 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:37.163240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:37.163243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:37.163493 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:37.164545 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:37.164552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:37.164926 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 11:50:37.168196 (+ 3270us) client_negotiation.cc:240] Negotiation successful
0924 11:50:37.168685 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19977,"thread_start_us":19712,"threads_started":1}
I20260924 11:50:37.173553 16440 heartbeater.cc:347] Connected to a master server at 127.1.210.62:39227
I20260924 11:50:37.173924 16440 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:37.174853 16440 heartbeater.cc:511] Master 127.1.210.62:39227 requested a full tablet report, sending...
I20260924 11:50:37.177325 16255 ts_manager.cc:194] Registered new tserver with Master: f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1:40043)
I20260924 11:50:37.178903 16255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:50519
W20260924 11:50:37.506783 16444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:37.507091 16444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:37.507259 16444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:37.514438 16444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:37.514573 16444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:37.514745 16444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:50:37.524156 16444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:39417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=33039
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:37.525450 16444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:37.526858 16444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:37.542251 16449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:37.542132 16450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:37.542356 16452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:37.543556 16444 server_base.cc:1034] running on GCE node
I20260924 11:50:37.544191 16444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:37.545111 16444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:37.546372 16444 hybrid_clock.cc:648] HybridClock initialized: now 1790250637546306 us; error 51 us; skew 500 ppm
I20260924 11:50:37.546753 16444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:37 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:37.550161 16444 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:50:37.551647 16444 webserver.cc:460] Webserver started at http://127.1.210.2:33039/ using document root <none> and password file <none>
I20260924 11:50:37.552428 16444 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:37.552573 16444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:37.557773 16444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 11:50:37.560350 16459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:37.561654 16444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:50:37.561821 16444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "4ff777fc0add435faedc5d9ce3894768"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "99888fd41b7610403b90ec3c3aed6b97"
server_key_iv: "bf9970912f73d0f73c4943e65ae014dd"
server_key_version: "encryptionkey@0"
I20260924 11:50:37.562775 16444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:37.590219 16444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:37.594843 16444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:37.595119 16444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:37.596132 16444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:37.597463 16444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:37.597579 16444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:37.597707 16444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:37.597797 16444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:37.630358 16444 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39417
I20260924 11:50:37.630441 16571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39417 every 8 connection(s)
I20260924 11:50:37.631779 16444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:50:37.635190 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16444
I20260924 11:50:37.635969 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:43857
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:39227
--webserver_port=45199
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:50:37 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:37.654003 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:37.634381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:44143 (local address 127.1.210.62:39227)
0924 11:50:37.634588 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:37.634594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:37.635939 (+ 1345us) server_negotiation.cc:407] Connection header received
0924 11:50:37.638884 (+ 2945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:37.638904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:37.639002 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:37.639088 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:37.641220 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.641902 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.644158 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.644597 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.644722 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:37.648087 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:37.648092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:37.648095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:37.648126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:37.650768 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:37.651494 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:37.651499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:37.651503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:37.651631 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:37.652050 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:37.652056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:37.652059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:37.652420 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:37.652572 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:37.653415 (+ 843us) server_negotiation.cc:299] Negotiation successful
0924 11:50:37.653654 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:50:37.654935 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:37.634906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:44143)
0924 11:50:37.635752 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:37.635802 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:37.638596 (+ 2794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:37.639348 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:37.639370 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:37.640148 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:37.640993 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:37.641038 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.642059 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.642067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:37.644007 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:37.644019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:37.644822 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:37.644829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:37.645049 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:37.645781 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:37.645808 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:37.647865 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:37.650931 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:37.650940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:37.650956 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:37.651370 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:37.651737 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:37.651742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:37.651746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:37.651932 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:37.652564 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:37.652572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:37.652975 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 11:50:37.653317 (+ 342us) client_negotiation.cc:240] Negotiation successful
0924 11:50:37.653850 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260924 11:50:37.656654 16572 heartbeater.cc:347] Connected to a master server at 127.1.210.62:39227
I20260924 11:50:37.656960 16572 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:37.657687 16572 heartbeater.cc:511] Master 127.1.210.62:39227 requested a full tablet report, sending...
I20260924 11:50:37.659233 16255 ts_manager.cc:194] Registered new tserver with Master: 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417)
I20260924 11:50:37.660542 16255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:44143
W20260924 11:50:37.959630 16575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:37.959916 16575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:37.960013 16575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:37.967379 16575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:37.967481 16575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:37.967590 16575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:50:37.977290 16575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=45199
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:37.978586 16575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:37.979854 16575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:37.992592 16580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:37.992556 16583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:37.992625 16581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:37.993629 16575 server_base.cc:1034] running on GCE node
I20260924 11:50:37.994203 16575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:37.995123 16575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:37.996354 16575 hybrid_clock.cc:648] HybridClock initialized: now 1790250637996308 us; error 56 us; skew 500 ppm
I20260924 11:50:37.996656 16575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:37 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:37.999719 16575 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:50:38.001190 16575 webserver.cc:460] Webserver started at http://127.1.210.3:45199/ using document root <none> and password file <none>
I20260924 11:50:38.001870 16575 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:38.002018 16575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:38.007084 16575 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 11:50:38.009691 16590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:38.011006 16575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:38.011168 16575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "0a05841940da42c0922ae62c0d6fc9d6"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "d3784dca7c18fc68a6a9edff7d3e6257"
server_key_iv: "13bbef77c0103adfd5db5c31ebe54eb9"
server_key_version: "encryptionkey@0"
I20260924 11:50:38.012144 16575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:38.032225 16575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:38.036357 16575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:38.036621 16575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:38.037585 16575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:38.038938 16575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:38.039062 16575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:38.039187 16575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:38.039268 16575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:38.071674 16575 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:43857
I20260924 11:50:38.071755 16702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:43857 every 8 connection(s)
I20260924 11:50:38.073213 16575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:50:38.079404 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16575
Sep 24 11:50:38 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:38.095570 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.075708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:37101 (local address 127.1.210.62:39227)
0924 11:50:38.075885 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.075889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.076985 (+ 1096us) server_negotiation.cc:407] Connection header received
0924 11:50:38.078814 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.078820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.078909 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.079038 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:38.080722 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.081660 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.083887 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.084528 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.084609 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.088677 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:38.088683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:38.088686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:38.088716 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:38.092394 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.093082 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.093088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.093092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.093245 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.093703 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.093706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.093708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.094045 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:38.094169 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.094965 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.095177 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:50:38.095778 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.080250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.1.210.62:39227)
0924 11:50:38.080727 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.080731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.081187 (+ 456us) server_negotiation.cc:407] Connection header received
0924 11:50:38.081388 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.081392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.081448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.081561 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:38.083455 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.084376 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.086330 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.086780 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.086872 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.089840 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:38.089847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:38.089852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:38.089908 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:38.093055 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.093887 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.093891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.093894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.094027 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.094409 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.094412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.094414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.094638 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:38.094849 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.095174 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.095375 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"spinlock_wait_cycles":1408,"thread_start_us":250,"threads_started":1}
I20260924 11:50:38.097015 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.076175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:37101)
0924 11:50:38.076837 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.076874 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:38.078552 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:38.079272 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:38.079280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:38.079824 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:38.080512 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.080542 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.081818 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.081824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.083758 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.083769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.084743 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.084750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.085038 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.085731 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:38.085764 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:38.088479 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:38.092520 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:38.092526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:38.092538 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:38.092954 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:38.093350 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:38.093355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:38.093358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:38.093584 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:38.094172 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:38.094182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:38.094576 (+ 394us) client_negotiation.cc:769] Sending connection context
0924 11:50:38.094881 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 11:50:38.095408 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":260,"threads_started":1}
I20260924 11:50:38.098436 16703 heartbeater.cc:347] Connected to a master server at 127.1.210.62:39227
I20260924 11:50:38.098734 16703 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:38.099426 16703 heartbeater.cc:511] Master 127.1.210.62:39227 requested a full tablet report, sending...
I20260924 11:50:38.100715 16255 ts_manager.cc:194] Registered new tserver with Master: 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
I20260924 11:50:38.101557 16255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:37101
I20260924 11:50:38.110354 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:50:38.117998 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.111240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.1.210.62:39227)
0924 11:50:38.111443 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.111452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.111781 (+ 329us) server_negotiation.cc:407] Connection header received
0924 11:50:38.111922 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.111927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.112093 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.112285 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:50:38.112676 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.113860 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.115440 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.116084 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.116201 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.116751 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:50:38.117383 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:50:38.117521 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.117581 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.117659 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:50:38.118939 16255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:50:38.126087 16255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:50:38.146324 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.138238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35614)
0924 11:50:38.138920 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.138932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:38.139007 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:38.140103 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:38.140110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:38.140144 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:38.140600 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.140614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.142232 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.142238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.143652 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.143665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.145353 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.145359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.145561 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.145621 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 11:50:38.145764 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 11:50:38.145974 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":3031,"threads_started":1}
I20260924 11:50:38.147928 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.139039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.1.210.1:40043)
0924 11:50:38.139609 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.139615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.139638 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:50:38.139714 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.139719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.139911 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.140097 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:38.140845 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.142095 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.143800 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.145196 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.145328 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.145571 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.147377 (+ 1806us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.147568 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":2008,"threads_started":1}
I20260924 11:50:38.152091 16375 tablet_service.cc:1467] Processing CreateTablet for tablet 2134561fbd6c4976ba54be025b67b380 (DEFAULT_TABLE table=test-table [id=de7ba67349cb44689a16d67b579cd8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:38.153606 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.140220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43857 (local address 127.0.0.1:34460)
0924 11:50:38.140816 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.140831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:38.140945 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:38.146780 (+ 5835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:38.146784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:38.146812 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:38.147183 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.147196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.149260 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.149266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.150714 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.150727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.152683 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.152689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.152964 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.153003 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:50:38.153135 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:50:38.153275 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":1390,"threads_started":1}
I20260924 11:50:38.153687 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.140857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.1.210.3:43857)
0924 11:50:38.146071 (+ 5214us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.146083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.146118 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:50:38.146216 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.146224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.146512 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.146753 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:38.147379 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.149134 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.150869 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.152149 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.152272 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.152513 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.153208 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.153395 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5051,"thread_start_us":4907,"threads_started":1}
I20260924 11:50:38.153760 16375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2134561fbd6c4976ba54be025b67b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:38.155270 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.137584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39417 (local address 127.0.0.1:60486)
0924 11:50:38.138420 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.138459 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:38.138647 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:38.148004 (+ 9357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:38.148009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:38.148032 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:38.148390 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.148404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.150626 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.150631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.152713 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.152729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.154543 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.154550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.154711 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.154753 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 11:50:38.154841 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:50:38.154966 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":1682,"threads_started":1}
I20260924 11:50:38.155507 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.137884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.1.210.2:39417)
0924 11:50:38.147420 (+ 9536us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.147429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.147462 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:50:38.147557 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.147564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.147796 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.148563 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:38.148926 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.150473 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.152903 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.154341 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.154490 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.154714 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.154899 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.155061 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9353,"thread_start_us":9201,"threads_started":1}
I20260924 11:50:38.158211 16638 tablet_service.cc:1467] Processing CreateTablet for tablet 2134561fbd6c4976ba54be025b67b380 (DEFAULT_TABLE table=test-table [id=de7ba67349cb44689a16d67b579cd8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:38.159444 16638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2134561fbd6c4976ba54be025b67b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:38.160413 16507 tablet_service.cc:1467] Processing CreateTablet for tablet 2134561fbd6c4976ba54be025b67b380 (DEFAULT_TABLE table=test-table [id=de7ba67349cb44689a16d67b579cd8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:38.161710 16507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2134561fbd6c4976ba54be025b67b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:38.169375 16719 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Bootstrap starting.
I20260924 11:50:38.172549 16720 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Bootstrap starting.
I20260924 11:50:38.174067 16719 tablet_bootstrap.cc:654] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:38.175215 16719 log.cc:824] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:38.176040 16720 tablet_bootstrap.cc:654] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:38.177678 16720 log.cc:824] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:38.178337 16719 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: No bootstrap required, opened a new log
I20260924 11:50:38.178630 16719 ts_tablet_manager.cc:1397] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260924 11:50:38.180500 16720 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: No bootstrap required, opened a new log
I20260924 11:50:38.180742 16720 ts_tablet_manager.cc:1397] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:50:38.182031 16440 heartbeater.cc:503] Master 127.1.210.62:39227 was elected leader, sending a full tablet report...
I20260924 11:50:38.183857 16722 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Bootstrap starting.
I20260924 11:50:38.187510 16722 tablet_bootstrap.cc:654] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:38.188333 16719 raft_consensus.cc:348] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.188817 16719 raft_consensus.cc:374] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:38.188795 16722 log.cc:824] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:38.188941 16719 raft_consensus.cc:729] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c8544e9b7c45409e38ae5d59f0f7de, State: Initialized, Role: FOLLOWER
I20260924 11:50:38.189620 16719 consensus_queue.cc:260] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.189989 16720 raft_consensus.cc:348] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.190409 16720 raft_consensus.cc:374] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:38.190506 16720 raft_consensus.cc:729] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a05841940da42c0922ae62c0d6fc9d6, State: Initialized, Role: FOLLOWER
I20260924 11:50:38.191053 16720 consensus_queue.cc:260] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.191498 16719 ts_tablet_manager.cc:1428] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 11:50:38.192443 16703 heartbeater.cc:503] Master 127.1.210.62:39227 was elected leader, sending a full tablet report...
I20260924 11:50:38.193037 16720 ts_tablet_manager.cc:1428] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260924 11:50:38.193645 16722 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: No bootstrap required, opened a new log
I20260924 11:50:38.193914 16722 ts_tablet_manager.cc:1397] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 11:50:38.201277 16722 raft_consensus.cc:348] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.201597 16722 raft_consensus.cc:374] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:38.201664 16722 raft_consensus.cc:729] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff777fc0add435faedc5d9ce3894768, State: Initialized, Role: FOLLOWER
I20260924 11:50:38.202099 16722 consensus_queue.cc:260] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.203051 16572 heartbeater.cc:503] Master 127.1.210.62:39227 was elected leader, sending a full tablet report...
I20260924 11:50:38.203634 16722 ts_tablet_manager.cc:1428] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260924 11:50:38.324865 16704 tablet.cc:2369] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:50:38.376750 16441 tablet.cc:2369] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:50:38.383703 16573 tablet.cc:2369] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:50:38.462157 16725 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:38.462457 16725 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.464105 16725 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
I20260924 11:50:38.471309 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.464495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:54553 (local address 127.1.210.2:39417)
0924 11:50:38.464676 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.464681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.465264 (+ 583us) server_negotiation.cc:407] Connection header received
0924 11:50:38.465491 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.465497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.465593 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.465720 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:38.466354 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.467371 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.469395 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.470334 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.470430 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.470504 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.470921 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.471061 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:50:38.471390 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.464442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39417 (local address 127.1.210.1:54553)
0924 11:50:38.465160 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.465181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:38.465330 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:38.465787 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:38.465791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:38.465821 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:38.466189 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.466215 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.467494 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.467500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.469237 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.469254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.470531 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.470538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.470714 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.470751 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:50:38.470858 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 11:50:38.471013 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1,"thread_start_us":376,"threads_started":1}
I20260924 11:50:38.471997 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.465446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:38401 (local address 127.1.210.3:43857)
0924 11:50:38.465706 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.465713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.466144 (+ 431us) server_negotiation.cc:407] Connection header received
0924 11:50:38.466284 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.466292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.466392 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.466467 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:38.467085 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.468030 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.469910 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.470857 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.470971 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.471061 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.471604 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.471751 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:50:38.472496 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.465318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43857 (local address 127.1.210.1:38401)
0924 11:50:38.466039 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.466055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:38.466158 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:38.466507 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:38.466512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:38.466535 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:38.466933 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.466945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.468166 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.468170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.469753 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:38.469767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.471059 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.471065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:38.471247 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.471282 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:50:38.472171 (+ 889us) client_negotiation.cc:240] Negotiation successful
0924 11:50:38.472272 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":367,"threads_started":1}
I20260924 11:50:38.472443 16527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2134561fbd6c4976ba54be025b67b380" candidate_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff777fc0add435faedc5d9ce3894768" is_pre_election: true
I20260924 11:50:38.473032 16527 raft_consensus.cc:2454] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9c8544e9b7c45409e38ae5d59f0f7de in term 0.
I20260924 11:50:38.473495 16658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2134561fbd6c4976ba54be025b67b380" candidate_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a05841940da42c0922ae62c0d6fc9d6" is_pre_election: true
I20260924 11:50:38.473773 16329 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff777fc0add435faedc5d9ce3894768, f9c8544e9b7c45409e38ae5d59f0f7de; no voters:
I20260924 11:50:38.474050 16658 raft_consensus.cc:2454] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9c8544e9b7c45409e38ae5d59f0f7de in term 0.
I20260924 11:50:38.474232 16725 raft_consensus.cc:2781] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:50:38.474390 16725 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:50:38.474504 16725 raft_consensus.cc:3037] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:38.476998 16725 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.477614 16725 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 1 election: Requested vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
I20260924 11:50:38.478009 16527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2134561fbd6c4976ba54be025b67b380" candidate_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff777fc0add435faedc5d9ce3894768"
I20260924 11:50:38.478106 16658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2134561fbd6c4976ba54be025b67b380" candidate_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a05841940da42c0922ae62c0d6fc9d6"
I20260924 11:50:38.478204 16527 raft_consensus.cc:3037] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:38.478327 16658 raft_consensus.cc:3037] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:38.480345 16527 raft_consensus.cc:2454] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9c8544e9b7c45409e38ae5d59f0f7de in term 1.
I20260924 11:50:38.480470 16658 raft_consensus.cc:2454] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9c8544e9b7c45409e38ae5d59f0f7de in term 1.
I20260924 11:50:38.480700 16329 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff777fc0add435faedc5d9ce3894768, f9c8544e9b7c45409e38ae5d59f0f7de; no voters:
I20260924 11:50:38.480988 16725 raft_consensus.cc:2781] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:38.481532 16725 raft_consensus.cc:686] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 LEADER]: Becoming Leader. State: Replica: f9c8544e9b7c45409e38ae5d59f0f7de, State: Running, Role: LEADER
I20260924 11:50:38.481971 16725 consensus_queue.cc:237] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:38.486451 16254 catalog_manager.cc:5495] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1). New cstate: current_term: 1 leader_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } health_report { overall_health: HEALTHY } } }
I20260924 11:50:38.516309 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.511380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.1.210.1:40043)
0924 11:50:38.511560 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.511564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.511582 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:50:38.511723 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.511727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.511802 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.511937 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:50:38.512221 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.512928 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.514376 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.514819 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.514922 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.515356 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:50:38.515831 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:50:38.515964 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.516057 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.516119 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:50:38.524257 16658 raft_consensus.cc:1261] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Refusing update from remote peer f9c8544e9b7c45409e38ae5d59f0f7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:38.524330 16527 raft_consensus.cc:1261] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Refusing update from remote peer f9c8544e9b7c45409e38ae5d59f0f7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:50:38.525257 16725 consensus_queue.cc:1035] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:38.525676 16730 consensus_queue.cc:1035] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:50:38.536353 16733 mvcc.cc:205] Tried to move back new op lower bound from 7332866615385444352 to 7332866615225344000. Current Snapshot: MvccSnapshot[applied={T|T < 7332866615385444352}]
I20260924 11:50:38.536940 16735 mvcc.cc:205] Tried to move back new op lower bound from 7332866615385444352 to 7332866615225344000. Current Snapshot: MvccSnapshot[applied={T|T < 7332866615385444352}]
I20260924 11:50:38.553812 16254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:50:38.554097 16254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:50:38.557307 16254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Sending DeleteTablet for 3 replicas of tablet 2134561fbd6c4976ba54be025b67b380
I20260924 11:50:38.558516 16375 tablet_service.cc:1514] Processing DeleteTablet for tablet 2134561fbd6c4976ba54be025b67b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:38 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:35614
I20260924 11:50:38.558527 16507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2134561fbd6c4976ba54be025b67b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:38 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:60486
I20260924 11:50:38.558691 16638 tablet_service.cc:1514] Processing DeleteTablet for tablet 2134561fbd6c4976ba54be025b67b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:50:38 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:34460
I20260924 11:50:38.559470 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16292
I20260924 11:50:38.571004 16744 tablet_replica.cc:330] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: stopping tablet replica
I20260924 11:50:38.571587 16744 raft_consensus.cc:2229] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:50:38.572220 16744 raft_consensus.cc:2258] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:38.573189 16742 tablet_replica.cc:330] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: stopping tablet replica
I20260924 11:50:38.573695 16742 raft_consensus.cc:2229] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:50:38.574265 16742 raft_consensus.cc:2258] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:38.576918 16742 ts_tablet_manager.cc:1905] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:38.581764 16743 tablet_replica.cc:330] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: stopping tablet replica
I20260924 11:50:38.582173 16743 raft_consensus.cc:2229] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:50:38.582760 16743 raft_consensus.cc:2258] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:50:38.583906 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.560994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.1.210.1:40043)
0924 11:50:38.561194 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.561200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.561216 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:50:38.561273 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.561278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.561355 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.561466 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:38.563373 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.564642 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.566824 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.567575 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.567716 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.574097 (+ 6381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:38.574112 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:38.574129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:38.574181 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:38.579086 (+ 4905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.580025 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.580035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.580038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.580218 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.580666 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.580672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.580675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.581682 (+ 1007us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:38.582068 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.583256 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.583509 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:50:38.586894 16742 ts_tablet_manager.cc:1918] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:38.586061 16743 ts_tablet_manager.cc:1905] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:38.587061 16742 log.cc:1196] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/2134561fbd6c4976ba54be025b67b380
I20260924 11:50:38.587162 16435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:38.588088 16742 ts_tablet_manager.cc:1939] T 2134561fbd6c4976ba54be025b67b380 P 4ff777fc0add435faedc5d9ce3894768: Deleting consensus metadata
I20260924 11:50:38.588779 16744 ts_tablet_manager.cc:1905] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:50:38.590391 16240 catalog_manager.cc:4841] TS 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): tablet 2134561fbd6c4976ba54be025b67b380 (table test-table [id=de7ba67349cb44689a16d67b579cd8db]) successfully deleted
I20260924 11:50:38.595626 16744 ts_tablet_manager.cc:1918] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:50:38.595786 16744 log.cc:1196] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/2134561fbd6c4976ba54be025b67b380
I20260924 11:50:38.596256 16744 ts_tablet_manager.cc:1939] T 2134561fbd6c4976ba54be025b67b380 P 0a05841940da42c0922ae62c0d6fc9d6: Deleting consensus metadata
I20260924 11:50:38.598013 16240 catalog_manager.cc:4841] TS 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): tablet 2134561fbd6c4976ba54be025b67b380 (table test-table [id=de7ba67349cb44689a16d67b579cd8db]) successfully deleted
I20260924 11:50:38.686458 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16292
I20260924 11:50:38.705024 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16444
W20260924 11:50:38.705169 16242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.1:40043 (error 108)
I20260924 11:50:38.707003 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.706327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35642)
0924 11:50:38.706725 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.706845 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":186,"threads_started":1}
W20260924 11:50:38.707245 16242 catalog_manager.cc:4584] TS f9c8544e9b7c45409e38ae5d59f0f7de (127.1.210.1:40043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2134561fbd6c4976ba54be025b67b380: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
I20260924 11:50:38.718464 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.705889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.1.210.2:39417)
0924 11:50:38.706066 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.706072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.706167 (+ 95us) server_negotiation.cc:407] Connection header received
0924 11:50:38.706379 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.706385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.706493 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.706613 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:38.708094 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.708803 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.710130 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.710532 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.710624 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.713023 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:38.713097 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:38.713119 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:38.713158 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:38.715766 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.716422 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.716429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.716432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.716553 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.716934 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.716943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.716948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.717488 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:38.717791 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.717998 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.718171 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:50:38.719249 16567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:38.761103 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.760539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35658)
0924 11:50:38.760777 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.760900 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 11:50:38.762228 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.761821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35672)
0924 11:50:38.761974 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.762059 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:50:38.809496 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16444
I20260924 11:50:38.822700 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.822088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35680)
0924 11:50:38.822362 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.822494 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:50:38.823860 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.823403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35682)
0924 11:50:38.823586 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.823681 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:50:38.826967 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16575
I20260924 11:50:38.844036 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.832056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.1.210.3:43857)
0924 11:50:38.832211 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:38.832215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:38.832233 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:50:38.832390 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:38.832394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:38.832472 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:38.832570 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:38.833878 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.834580 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.835882 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:38.836228 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:38.836317 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:38.838743 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:38.838772 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:38.838787 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:38.838816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:38.841821 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.842427 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.842434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.842436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.842549 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:38.842851 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:38.842854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:38.842856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:38.843180 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:38.843389 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:38.843620 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 11:50:38.843771 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:50:38.844702 16698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:38.904577 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.903957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35694)
0924 11:50:38.904270 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.904393 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 11:50:38.905730 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:38.905286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40043 (local address 127.0.0.1:35700)
0924 11:50:38.905499 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:38.905576 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 11:50:38.935933 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16575
I20260924 11:50:38.952297 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16222
I20260924 11:50:38.953459 16285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:39.066536 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16222
I20260924 11:50:39.088099 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39227
--webserver_interface=127.1.210.62
--webserver_port=45551
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:39.422083 16751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:39.422389 16751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:39.422546 16751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:39.429595 16751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:39.429737 16751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:39.429836 16751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:39.429932 16751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:39.430013 16751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:39.439535 16751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=45551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:39.440822 16751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:39.442080 16751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:39.453308 16756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:39.453434 16759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:39.453344 16757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:39.454032 16751 server_base.cc:1034] running on GCE node
I20260924 11:50:39.454687 16751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:39.455927 16751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:39.457155 16751 hybrid_clock.cc:648] HybridClock initialized: now 1790250639457112 us; error 36 us; skew 500 ppm
I20260924 11:50:39.457489 16751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:39 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:39.460470 16751 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:39.461908 16751 webserver.cc:460] Webserver started at http://127.1.210.62:45551/ using document root <none> and password file <none>
I20260924 11:50:39.462543 16751 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:39.462670 16751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:39.467829 16751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 11:50:39.470585 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:39.472009 16751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:50:39.472191 16751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "329d886e40374bacaa6c39bd22c3c6a7"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "600e04715db2c3e8198c40877078b653"
server_key_iv: "ec5db06e2b60e8c1bbde3e710b2d3fc8"
server_key_version: "encryptionkey@0"
I20260924 11:50:39.473213 16751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:39.493638 16751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:39.497774 16751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:39.498046 16751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:39.515823 16751 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:39227
I20260924 11:50:39.515842 16817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:39227 every 8 connection(s)
I20260924 11:50:39.517320 16751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:39.519763 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16751
I20260924 11:50:39.520654 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:40043
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:39227
--webserver_port=42019
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:39.524137 16818 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:50:39.527300 16818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap starting.
I20260924 11:50:39.548883 16818 log.cc:824] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:39.564205 16818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:50:39.564708 16818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7: Bootstrap complete.
I20260924 11:50:39.573765 16818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:39.574680 16818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329d886e40374bacaa6c39bd22c3c6a7, State: Initialized, Role: FOLLOWER
I20260924 11:50:39.575238 16818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:39.575461 16818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:39.575584 16818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:39.575814 16818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:50:39.579710 16818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:39.580193 16818 leader_election.cc:302] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 329d886e40374bacaa6c39bd22c3c6a7; no voters:
I20260924 11:50:39.580859 16818 leader_election.cc:288] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 11:50:39.580997 16822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:50:39.583166 16818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:39.588117 16822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [term 3 LEADER]: Becoming Leader. State: Replica: 329d886e40374bacaa6c39bd22c3c6a7, State: Running, Role: LEADER
I20260924 11:50:39.588807 16822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } }
I20260924 11:50:39.591610 16823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "329d886e40374bacaa6c39bd22c3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } } }
I20260924 11:50:39.591990 16823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:39.591811 16824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329d886e40374bacaa6c39bd22c3c6a7. Latest consensus state: current_term: 3 leader_uuid: "329d886e40374bacaa6c39bd22c3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329d886e40374bacaa6c39bd22c3c6a7" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39227 } } }
I20260924 11:50:39.592201 16824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329d886e40374bacaa6c39bd22c3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:39.602473 16832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:39.612850 16832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:39.615082 16832 catalog_manager.cc:1259] Loaded cluster ID: a8e3df71bdd04f88bb00d3f218d61055
I20260924 11:50:39.615275 16832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 11:50:39.617856 16832 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1faf15d9ad google::LogMessage::Fail() at ??:0
@ 0x7f1faf16088a google::LogMessage::SendToLog() at ??:0
@ 0x7f1faf15d4de google::LogMessage::Flush() at ??:0
@ 0x7f1faf15ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1fc4b90512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1fc4b8d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1fc4c15610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1fc4c15439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1fb1da9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1fb1dac91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1fb1dac799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1fb1d87eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1fb50616db start_thread at ??:0
@ 0x7f1fadff761f clone at ??:0
W20260924 11:50:39.904259 16820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:39.904562 16820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:39.904716 16820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:39.911967 16820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:39.912071 16820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:39.912235 16820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:50:39.921941 16820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:40043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=42019
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:39.923207 16820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:39.924480 16820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:39.937049 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:39.937143 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:39.937013 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:39.938221 16820 server_base.cc:1034] running on GCE node
I20260924 11:50:39.938794 16820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:39.939672 16820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:39.940886 16820 hybrid_clock.cc:648] HybridClock initialized: now 1790250639940832 us; error 49 us; skew 500 ppm
I20260924 11:50:39.941221 16820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:39 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:39.944183 16820 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:50:39.945639 16820 webserver.cc:460] Webserver started at http://127.1.210.1:42019/ using document root <none> and password file <none>
I20260924 11:50:39.946293 16820 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:39.946429 16820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:39.951424 16820 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:50:39.954207 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:39.955415 16820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:39.955577 16820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "f9c8544e9b7c45409e38ae5d59f0f7de"
format_stamp: "Formatted at 2026-09-24 11:50:33 on dist-test-slave-dd32"
server_key: "07a07e6ab6c0fc89985934c71dd76353"
server_key_iv: "ae616d6bbc1642d92454a002d82199a6"
server_key_version: "encryptionkey@0"
I20260924 11:50:39.956504 16820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:39.976322 16820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:39.980386 16820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:39.980674 16820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:39.981793 16820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:39.983421 16877 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 11:50:39.988689 16820 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 11:50:39.988823 16820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 11:50:39.988946 16820 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 11:50:39.992043 16820 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 11:50:39.992138 16820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 11:50:39.992436 16877 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Bootstrap starting.
I20260924 11:50:40.000483 16877 log.cc:824] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:40.019069 16877 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:50:40.020051 16877 tablet_bootstrap.cc:492] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Bootstrap complete.
I20260924 11:50:40.020702 16877 ts_tablet_manager.cc:1397] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.004s
I20260924 11:50:40.030664 16820 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:40043
I20260924 11:50:40.030701 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:40043 every 8 connection(s)
I20260924 11:50:40.032305 16820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:50:40.032932 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16820
I20260924 11:50:40.033706 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:39417
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:39227
--webserver_port=33039
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:40.033533 16877 raft_consensus.cc:348] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:40.035389 16877 raft_consensus.cc:729] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c8544e9b7c45409e38ae5d59f0f7de, State: Initialized, Role: FOLLOWER
I20260924 11:50:40.036229 16877 consensus_queue.cc:260] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:40.048250 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.035409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:37541)
0924 11:50:40.047041 (+ 11632us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.047315 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11361,"threads_started":1}
W20260924 11:50:40.049726 16985 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:39227 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.62:39227: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
I20260924 11:50:40.050447 16877 ts_tablet_manager.cc:1428] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de: Time spent starting tablet: real 0.030s user 0.024s sys 0.008s
I20260924 11:50:40.050959 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.050594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:58021)
0924 11:50:40.050762 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.050834 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:50:40.051960 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.051562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:56325)
0924 11:50:40.051750 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.051820 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:50:40.052285 16985 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:50:40.371212 16989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:40.371503 16989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:40.371730 16989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:40.380520 16989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:40.380642 16989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:40.380823 16989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:50:40.391135 16989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:39417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=33039
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:40.392463 16989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:40.393894 16989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:40.407701 16994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:40.407646 16997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:40.407809 16995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:40.408557 16989 server_base.cc:1034] running on GCE node
I20260924 11:50:40.409211 16989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:40.410194 16989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:40.411418 16989 hybrid_clock.cc:648] HybridClock initialized: now 1790250640411360 us; error 46 us; skew 500 ppm
I20260924 11:50:40.411720 16989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:40 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250640, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:40.414871 16989 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:50:40.416282 16989 webserver.cc:460] Webserver started at http://127.1.210.2:33039/ using document root <none> and password file <none>
I20260924 11:50:40.416950 16989 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:40.417138 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:40.422499 16989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 11:50:40.425133 17004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:40.426365 16989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:50:40.426545 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "4ff777fc0add435faedc5d9ce3894768"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "99888fd41b7610403b90ec3c3aed6b97"
server_key_iv: "bf9970912f73d0f73c4943e65ae014dd"
server_key_version: "encryptionkey@0"
I20260924 11:50:40.427515 16989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:40.447669 16989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:40.451738 16989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:40.452010 16989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:40.453089 16989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:40.454564 16989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:40.454691 16989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:40.454809 16989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:40.454895 16989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:40.488776 16989 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39417
I20260924 11:50:40.488858 17116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39417 every 8 connection(s)
I20260924 11:50:40.490214 16989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:50:40.493481 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 16989
I20260924 11:50:40.494086 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:43857
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:39227
--webserver_port=45199
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:50:40.495142 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.493165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:37175)
0924 11:50:40.493975 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.494267 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":351,"threads_started":1}
W20260924 11:50:40.495757 17117 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:39227 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.62:39227: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
I20260924 11:50:40.496601 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.496296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:44767)
0924 11:50:40.496441 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.496505 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:50:40.497493 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.497203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:40051)
0924 11:50:40.497316 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.497392 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 11:50:40.497766 17117 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:50:40.813756 17120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:40.814010 17120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:40.814198 17120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:40.821382 17120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:40.821501 17120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:40.821668 17120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:50:40.831214 17120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=45199
--tserver_master_addrs=127.1.210.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:40.832418 17120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:40.833716 17120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:40.848515 17125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:40.848441 17128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:40.848469 17126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:40.855110 17120 server_base.cc:1034] running on GCE node
I20260924 11:50:40.855783 17120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:40.856781 17120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:40.857991 17120 hybrid_clock.cc:648] HybridClock initialized: now 1790250640857921 us; error 62 us; skew 500 ppm
I20260924 11:50:40.858330 17120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:40 dist-test-slave-dd32 krb5kdc[15670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250640, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:40.861735 17120 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:50:40.863250 17120 webserver.cc:460] Webserver started at http://127.1.210.3:45199/ using document root <none> and password file <none>
I20260924 11:50:40.863982 17120 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:40.864110 17120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:40.869429 17120 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:50:40.871938 17135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:40.873220 17120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:40.873397 17120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "0a05841940da42c0922ae62c0d6fc9d6"
format_stamp: "Formatted at 2026-09-24 11:50:34 on dist-test-slave-dd32"
server_key: "d3784dca7c18fc68a6a9edff7d3e6257"
server_key_iv: "13bbef77c0103adfd5db5c31ebe54eb9"
server_key_version: "encryptionkey@0"
I20260924 11:50:40.874326 17120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:40.897531 17120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:40.901187 17120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:40.901448 17120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:40.902352 17120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:40.903610 17120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:40.903734 17120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:40.903848 17120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:40.903931 17120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:40.941999 17120 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:43857
I20260924 11:50:40.942062 17247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:43857 every 8 connection(s)
I20260924 11:50:40.943439 17120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:50:40.946538 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17120
I20260924 11:50:40.948675 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.946419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:40299)
0924 11:50:40.947277 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.947599 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
W20260924 11:50:40.949414 17248 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:39227 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.62:39227: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
I20260924 11:50:40.950173 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.949877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:54747)
0924 11:50:40.950009 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.950076 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 11:50:40.950969 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:40.950656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:45447)
0924 11:50:40.950790 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:40.950866 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:50:40.951175 17248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:50:41.053555 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.052741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:60849)
0924 11:50:41.053222 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:41.053310 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":215,"threads_started":1}
I20260924 11:50:41.366683 17253 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:41.366979 17253 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:41.368570 17253 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
Sep 24 11:50:41 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
Sep 24 11:50:41 dist-test-slave-dd32 krb5kdc[15670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:50:41.389525 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.369363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43857 (local address 127.1.210.1:35863)
0924 11:50:41.370221 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:41.370240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:41.370356 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:41.372731 (+ 2375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:41.372736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:41.373523 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:41.374120 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:41.374145 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.375757 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.375763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:41.377406 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:41.377418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.378355 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.378364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:41.378668 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:41.379469 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:41.379486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:41.381935 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:41.385687 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:41.385697 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:41.385711 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:41.386290 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:41.386795 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:41.386799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:41.386803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:41.387057 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:41.387964 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:41.387969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:41.388215 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 11:50:41.388514 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 11:50:41.389028 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":50,"thread_start_us":644,"threads_started":1}
I20260924 11:50:41.389601 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.369891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:35863 (local address 127.1.210.3:43857)
0924 11:50:41.370538 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:41.370548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:41.370580 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:50:41.372136 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:41.372143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:41.372483 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:41.372770 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:41.374348 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.375525 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.377531 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.378172 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.378306 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:41.382217 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:41.382227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:41.382244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:41.382291 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:41.385559 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:41.386438 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:41.386446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:41.386453 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:41.386651 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:41.387193 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:41.387199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:41.387203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:41.387833 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:41.388019 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:41.388738 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 11:50:41.389179 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":348,"threads_started":1}
I20260924 11:50:41.389717 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.369132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:48895 (local address 127.1.210.2:39417)
0924 11:50:41.369874 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:41.369892 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:41.369943 (+ 51us) server_negotiation.cc:407] Connection header received
0924 11:50:41.371591 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:41.371600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:41.371970 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:41.372303 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:41.374451 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.376072 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.377600 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.378282 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.378452 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:41.382539 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:41.382545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:41.382557 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:41.382590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:41.386121 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:41.387089 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:41.387099 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:41.387137 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:41.387363 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:41.387713 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:41.387718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:41.387721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:41.388265 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:41.388399 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:41.388947 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 11:50:41.389322 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":465,"threads_started":1}
I20260924 11:50:41.389544 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.368934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39417 (local address 127.1.210.1:48895)
0924 11:50:41.369134 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:41.369171 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:41.369736 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:41.372433 (+ 2697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:41.372446 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:41.373226 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:41.374119 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:41.374140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.376227 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.376231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:41.377431 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:41.377444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:41.378509 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:41.378515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:41.378689 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:41.379429 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:41.379452 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:41.382295 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:41.386306 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:41.386313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:41.386319 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:41.386925 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:41.387471 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:41.387475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:41.387477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:41.387598 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:41.388386 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:41.388390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:41.388558 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 11:50:41.388679 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 11:50:41.389196 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":305}
W20260924 11:50:41.392899 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
W20260924 11:50:41.393292 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
I20260924 11:50:41.393426 16872 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9c8544e9b7c45409e38ae5d59f0f7de; no voters: 0a05841940da42c0922ae62c0d6fc9d6, 4ff777fc0add435faedc5d9ce3894768
I20260924 11:50:41.393745 17253 raft_consensus.cc:2726] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:50:41.499049 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.498211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:35645)
0924 11:50:41.498660 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:41.498804 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
I20260924 11:50:41.952333 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:41.951660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:56389)
0924 11:50:41.952047 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:41.952145 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":189,"threads_started":1}
I20260924 11:50:42.054976 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:42.054299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:34591)
0924 11:50:42.054733 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:42.054810 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":237,"threads_started":1}
I20260924 11:50:42.500600 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:42.499844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:39745)
0924 11:50:42.500283 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:42.500360 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":261,"threads_started":1}
I20260924 11:50:42.953953 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:42.953240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:35769)
0924 11:50:42.953667 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:42.953755 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260924 11:50:43.056663 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:43.055882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:44397)
0924 11:50:43.056375 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:43.056454 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":227,"threads_started":1}
I20260924 11:50:43.065996 17263 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:43.066190 17263 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:43.066761 17263 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
W20260924 11:50:43.067718 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
W20260924 11:50:43.067970 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
I20260924 11:50:43.068081 16872 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9c8544e9b7c45409e38ae5d59f0f7de; no voters: 0a05841940da42c0922ae62c0d6fc9d6, 4ff777fc0add435faedc5d9ce3894768
I20260924 11:50:43.068310 17263 raft_consensus.cc:2726] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:50:43.502301 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:43.501510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:49111)
0924 11:50:43.501988 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:43.502090 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":243,"threads_started":1}
I20260924 11:50:43.955466 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:43.954755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:56527)
0924 11:50:43.955205 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:43.955304 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260924 11:50:44.058279 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:44.057593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:39541)
0924 11:50:44.058027 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:44.058107 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":220,"threads_started":1}
I20260924 11:50:44.503893 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:44.503142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:36815)
0924 11:50:44.503642 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:44.503720 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":242,"threads_started":1}
I20260924 11:50:44.957195 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:44.956338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:36147)
0924 11:50:44.956848 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:44.956936 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":245,"threads_started":1}
I20260924 11:50:45.060204 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:45.059373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:52299)
0924 11:50:45.059932 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:45.060015 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":242,"threads_started":1}
I20260924 11:50:45.505642 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:45.504795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:54685)
0924 11:50:45.505386 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:45.505478 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":384,"threads_started":1}
I20260924 11:50:45.958752 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:45.958058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:49645)
0924 11:50:45.958506 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:45.958590 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260924 11:50:46.061676 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:46.060983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:37147)
0924 11:50:46.061439 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:46.061517 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":267,"threads_started":1}
I20260924 11:50:46.380227 17273 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:46.380407 17273 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:46.380962 17273 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
W20260924 11:50:46.381999 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
W20260924 11:50:46.382202 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
I20260924 11:50:46.382329 16872 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9c8544e9b7c45409e38ae5d59f0f7de; no voters: 0a05841940da42c0922ae62c0d6fc9d6, 4ff777fc0add435faedc5d9ce3894768
I20260924 11:50:46.382575 17273 raft_consensus.cc:2726] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:50:46.507107 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:46.506414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:46595)
0924 11:50:46.506868 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:46.506942 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260924 11:50:46.960374 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:46.959674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:33215)
0924 11:50:46.960126 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:46.960204 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":259,"threads_started":1}
I20260924 11:50:47.063481 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:47.062476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:36725)
0924 11:50:47.063084 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:47.063186 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":366,"threads_started":1}
I20260924 11:50:47.508790 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:47.508055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:33855)
0924 11:50:47.508538 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:47.508617 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":213,"threads_started":1}
I20260924 11:50:47.962248 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:47.961343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:47321)
0924 11:50:47.961869 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:47.961952 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":221,"threads_started":1}
I20260924 11:50:48.064960 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:48.064277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:37295)
0924 11:50:48.064725 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:48.064802 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":230,"threads_started":1}
I20260924 11:50:48.510367 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:48.509736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:56261)
0924 11:50:48.510125 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:48.510199 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":201,"threads_started":1}
I20260924 11:50:48.963862 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:48.963184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:49829)
0924 11:50:48.963609 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:48.963688 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260924 11:50:49.066484 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:49.065780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:43017)
0924 11:50:49.066208 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:49.066287 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":213,"threads_started":1}
I20260924 11:50:49.512092 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:49.511394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:58009)
0924 11:50:49.511860 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:49.511946 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":229,"threads_started":1}
I20260924 11:50:49.965576 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:49.964736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:46329)
0924 11:50:49.965316 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:49.965409 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":344,"threads_started":1}
I20260924 11:50:50.068313 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:50.067471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:54461)
0924 11:50:50.068002 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:50.068088 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":257,"threads_started":1}
I20260924 11:50:50.164587 17286 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:50.164819 17286 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:50.165548 17286 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
W20260924 11:50:50.166466 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
W20260924 11:50:50.166700 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
I20260924 11:50:50.166781 16872 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9c8544e9b7c45409e38ae5d59f0f7de; no voters: 0a05841940da42c0922ae62c0d6fc9d6, 4ff777fc0add435faedc5d9ce3894768
I20260924 11:50:50.167088 17286 raft_consensus.cc:2726] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:50:50.513819 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:50.513046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:51561)
0924 11:50:50.513541 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:50.513642 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":224,"threads_started":1}
I20260924 11:50:50.967144 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:50.966442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:42841)
0924 11:50:50.966881 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:50.966973 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260924 11:50:51.069960 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:51.069293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:53291)
0924 11:50:51.069710 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:51.069786 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":231,"threads_started":1}
I20260924 11:50:51.515506 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:51.514681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:52777)
0924 11:50:51.515196 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:51.515299 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
I20260924 11:50:51.968731 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:51.968031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:59977)
0924 11:50:51.968495 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:51.968578 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260924 11:50:52.071533 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:52.070782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:34643)
0924 11:50:52.071286 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:52.071375 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":311,"threads_started":1}
I20260924 11:50:52.517055 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:52.516346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:46649)
0924 11:50:52.516798 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:52.516883 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":234,"threads_started":1}
I20260924 11:50:52.970275 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:52.969615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:42243)
0924 11:50:52.970036 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:52.970110 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260924 11:50:53.073232 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:53.072473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:36569)
0924 11:50:53.072887 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:53.072994 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
I20260924 11:50:53.518568 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:53.517878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:58467)
0924 11:50:53.518310 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:53.518395 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260924 11:50:53.972132 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:53.971358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:39663)
0924 11:50:53.971842 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:53.971963 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260924 11:50:54.075001 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:54.074193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:47943)
0924 11:50:54.074692 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:54.074790 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260924 11:50:54.520336 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:54.519561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:55045)
0924 11:50:54.520054 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:54.520145 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":244,"threads_started":1}
I20260924 11:50:54.698536 17300 raft_consensus.cc:482] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:50:54.698776 17300 raft_consensus.cc:504] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff777fc0add435faedc5d9ce3894768" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39417 } } peers { permanent_uuid: "0a05841940da42c0922ae62c0d6fc9d6" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43857 } } peers { permanent_uuid: "f9c8544e9b7c45409e38ae5d59f0f7de" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40043 } }
I20260924 11:50:54.699427 17300 leader_election.cc:288] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417), 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857)
W20260924 11:50:54.700667 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a05841940da42c0922ae62c0d6fc9d6 (127.1.210.3:43857): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
W20260924 11:50:54.700883 16872 leader_election.cc:341] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ff777fc0add435faedc5d9ce3894768 (127.1.210.2:39417): Not found: Tablet not found: 2134561fbd6c4976ba54be025b67b380
I20260924 11:50:54.701010 16872 leader_election.cc:302] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9c8544e9b7c45409e38ae5d59f0f7de; no voters: 0a05841940da42c0922ae62c0d6fc9d6, 4ff777fc0add435faedc5d9ce3894768
I20260924 11:50:54.701328 17300 raft_consensus.cc:2726] T 2134561fbd6c4976ba54be025b67b380 P f9c8544e9b7c45409e38ae5d59f0f7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:50:54.973711 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:54.972977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:59695)
0924 11:50:54.973422 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:54.973521 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":250,"threads_started":1}
I20260924 11:50:55.076995 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:55.076228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.1:52133)
0924 11:50:55.076669 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:55.076780 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260924 11:50:55.522089 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:55.521384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.2:39787)
0924 11:50:55.521821 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:55.521923 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260924 11:50:55.955044 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16820
I20260924 11:50:55.975713 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:55.974914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39227 (local address 127.1.210.3:34081)
0924 11:50:55.975420 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:55.975537 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":279,"threads_started":1}
I20260924 11:50:55.976451 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:55.956060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.1.210.1:40043)
0924 11:50:55.956645 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:55.956655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:55.956686 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:50:55.956756 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:55.956765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:55.957034 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:55.957222 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:55.959621 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:55.960679 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:55.961992 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:55.962553 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:55.962648 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:55.970439 (+ 7791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:55.970451 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:55.970480 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:55.970519 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:55.973565 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:55.974233 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:55.974241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:55.974245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:55.974428 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:55.974879 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:55.974883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:55.974885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:55.975236 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:55.975555 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:55.975858 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:50:55.976134 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":269,"threads_started":1}
I20260924 11:50:55.977738 16980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:56.066696 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16820
I20260924 11:50:56.082957 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 16989
I20260924 11:50:56.096091 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:56.083770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.1.210.2:39417)
0924 11:50:56.084242 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:56.084247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:56.084263 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:50:56.084319 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:56.084323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:56.084401 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:56.084491 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:56.086021 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:56.086859 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:56.088052 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:56.088468 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:56.088559 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:56.091062 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:56.091067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:56.091070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:56.091101 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:56.093710 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:56.094351 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:56.094355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:56.094357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:56.094496 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:56.094813 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:56.094816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:56.094818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:56.095093 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:56.095284 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:56.095539 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 11:50:56.095700 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":268,"threads_started":1}
I20260924 11:50:56.096855 17112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:56.181210 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 16989
I20260924 11:50:56.199143 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17120
I20260924 11:50:56.213263 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:56.200153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.1.210.3:43857)
0924 11:50:56.200811 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:56.200816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:56.200832 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:50:56.200888 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:56.200891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:56.200961 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:56.201055 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:56.202563 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:56.203335 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:56.204673 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:56.205185 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:56.205320 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:56.208022 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:56.208028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:56.208031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:56.208063 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:56.210862 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:56.211453 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:56.211456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:56.211458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:56.211602 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:56.211895 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:56.211898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:56.211900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:56.212144 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:56.212352 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:56.212571 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 11:50:56.212781 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":356,"threads_started":1}
I20260924 11:50:56.213948 17243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:50:56.298614 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17120
2026-09-24T11:50:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24586 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:50:56 dist-test-slave-dd32 krb5kdc[17313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:50:56 dist-test-slave-dd32 krb5kdc[17313](info): set up 2 sockets
Sep 24 11:50:56 dist-test-slave-dd32 krb5kdc[17313](info): commencing operation
krb5kdc: starting...
W20260924 11:50:57.429723 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:50:57 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:50:57.665670 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41355
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:58.004011 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:58.004350 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:58.004515 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:58.011782 17326 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:50:58.011924 17326 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:50:58.012017 17326 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:50:58.012111 17326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:50:58.012219 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:58.012341 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:58.012429 17326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:50:58.012516 17326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:50:58.022444 17326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:58.023759 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:58.025116 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:58.036959 17331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:58.036981 17334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:58.037009 17332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:58.037947 17326 server_base.cc:1034] running on GCE node
I20260924 11:50:58.038713 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:58.040015 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:58.041236 17326 hybrid_clock.cc:648] HybridClock initialized: now 1790250658041167 us; error 50 us; skew 500 ppm
I20260924 11:50:58.041558 17326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:58 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:58.044549 17326 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:50:58.045969 17326 webserver.cc:460] Webserver started at http://127.1.210.62:43837/ using document root <none> and password file <none>
I20260924 11:50:58.046703 17326 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:58.046830 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:58.047147 17326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:58.050019 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "2667fc4a393c48529c9c3ad031a6fe0e"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "a9336c68f37225810e30eec8f586e28c"
server_key_iv: "25cdb024bf0c70421928bf9f2a28e223"
server_key_version: "encryptionkey@0"
I20260924 11:50:58.050770 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "2667fc4a393c48529c9c3ad031a6fe0e"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "a9336c68f37225810e30eec8f586e28c"
server_key_iv: "25cdb024bf0c70421928bf9f2a28e223"
server_key_version: "encryptionkey@0"
I20260924 11:50:58.055531 17326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:50:58.059075 17341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:58.060453 17326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:50:58.060629 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "2667fc4a393c48529c9c3ad031a6fe0e"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "a9336c68f37225810e30eec8f586e28c"
server_key_iv: "25cdb024bf0c70421928bf9f2a28e223"
server_key_version: "encryptionkey@0"
I20260924 11:50:58.060839 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:58.079993 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:58.083798 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:58.084055 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:58.101589 17326 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:41355
I20260924 11:50:58.101611 17392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:41355 every 8 connection(s)
I20260924 11:50:58.102909 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:50:58.106441 17393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:58.107463 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17326
I20260924 11:50:58.107681 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:50:58.108296 1864 external_mini_cluster.cc:1428] Setting key 83194642d9580fab241ac4e2dfacc8a6
I20260924 11:50:58.120299 17393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap starting.
Sep 24 11:50:58 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:58.123736 17393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:58.124897 17393 log.cc:824] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:58.128001 17393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: No bootstrap required, opened a new log
I20260924 11:50:58.131520 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:58.110327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.1.210.62:41355)
0924 11:50:58.111127 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:58.111141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:58.111186 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:50:58.112827 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:58.112853 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:58.113367 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:58.113794 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:58.115073 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:58.117028 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:58.118369 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:58.119383 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:58.119559 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:58.124039 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:58.124047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:58.124060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:58.124100 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:58.127326 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:58.128243 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:58.128249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:58.128254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:58.128409 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:58.128853 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:58.128856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:58.128861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:58.129425 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:58.129633 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:58.130160 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 11:50:58.130533 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260924 11:50:58.136601 17393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:50:58.136955 17393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:58.137060 17393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2667fc4a393c48529c9c3ad031a6fe0e, State: Initialized, Role: FOLLOWER
I20260924 11:50:58.137580 17393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:50:58.137814 17393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:50:58.137954 17393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:50:58.138116 17393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:50:58.140175 17393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:50:58.140645 17393 leader_election.cc:302] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2667fc4a393c48529c9c3ad031a6fe0e; no voters:
I20260924 11:50:58.141300 17393 leader_election.cc:288] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:50:58.141407 17398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:50:58.142180 17398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 LEADER]: Becoming Leader. State: Replica: 2667fc4a393c48529c9c3ad031a6fe0e, State: Running, Role: LEADER
I20260924 11:50:58.142756 17398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:50:58.143626 17393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:50:58.145711 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2667fc4a393c48529c9c3ad031a6fe0e. Latest consensus state: current_term: 1 leader_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } } }
I20260924 11:50:58.145802 17399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } } }
I20260924 11:50:58.146147 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:58.146190 17399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: This master's current role is: LEADER
I20260924 11:50:58.147722 17404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:50:58.154099 17404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:50:58.164248 17404 catalog_manager.cc:1347] Generated new cluster ID: 98619ab58ba742f59a5091981907a393
I20260924 11:50:58.164400 17404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:50:58.182751 17404 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:50:58.184159 17404 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:50:58.192138 17404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Generated new TSK 0
I20260924 11:50:58.193188 17404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:50:58.245359 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:41355
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:58.578780 17419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:58.579066 17419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:58.579162 17419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:58.586522 17419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:58.586635 17419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:58.586771 17419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:50:58.596534 17419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:58.597919 17419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:58.599193 17419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:58.612378 17425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:58.612406 17424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:58.612879 17427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:58.613349 17419 server_base.cc:1034] running on GCE node
I20260924 11:50:58.613904 17419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:58.614784 17419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:58.616015 17419 hybrid_clock.cc:648] HybridClock initialized: now 1790250658615976 us; error 51 us; skew 500 ppm
I20260924 11:50:58.616356 17419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:58 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:58.619372 17419 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:50:58.620847 17419 webserver.cc:460] Webserver started at http://127.1.210.1:39641/ using document root <none> and password file <none>
I20260924 11:50:58.621596 17419 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:58.621740 17419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:58.622056 17419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:58.624917 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "997973b2f58f4be2ad3308a17cc6c953"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "dd7172f1cf17d474e194dc012a55d777"
server_key_iv: "4ac01cb3d0f39912ce8f8a2cebbf8f8d"
server_key_version: "encryptionkey@0"
I20260924 11:50:58.625751 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "997973b2f58f4be2ad3308a17cc6c953"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "dd7172f1cf17d474e194dc012a55d777"
server_key_iv: "4ac01cb3d0f39912ce8f8a2cebbf8f8d"
server_key_version: "encryptionkey@0"
I20260924 11:50:58.630722 17419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:50:58.634136 17434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:58.635365 17419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:50:58.635543 17419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "997973b2f58f4be2ad3308a17cc6c953"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "dd7172f1cf17d474e194dc012a55d777"
server_key_iv: "4ac01cb3d0f39912ce8f8a2cebbf8f8d"
server_key_version: "encryptionkey@0"
I20260924 11:50:58.635744 17419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:58.651644 17419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:58.655465 17419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:58.655748 17419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:58.656713 17419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:58.658129 17419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:58.658202 17419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:58.658286 17419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:58.658331 17419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:58.690649 17419 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:46141
I20260924 11:50:58.690692 17546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:46141 every 8 connection(s)
I20260924 11:50:58.691933 17419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:50:58.697724 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17419
I20260924 11:50:58.697947 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:50:58.698594 1864 external_mini_cluster.cc:1428] Setting key f75b58dbe53dfe5ecbbef62b007ffd5d
Sep 24 11:50:58 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:58.711745 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:58.694425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:40111 (local address 127.1.210.62:41355)
0924 11:50:58.694995 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:58.694999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:58.695830 (+ 831us) server_negotiation.cc:407] Connection header received
0924 11:50:58.697697 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:58.697703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:58.697821 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:58.697957 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:58.699553 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:58.700334 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:58.701976 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:58.702381 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:58.702517 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:58.705908 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:58.705913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:58.705916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:58.705946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:58.708673 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:58.709486 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:58.709496 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:58.709498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:58.709616 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:58.710046 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:58.710051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:58.710056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:58.710318 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:58.710476 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:58.711164 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 11:50:58.711388 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260924 11:50:58.712684 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:58.694977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:40111)
0924 11:50:58.695669 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:58.695710 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:58.697432 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:58.698199 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:58.698206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:58.698728 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:58.699298 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:58.699315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:58.700524 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:58.700528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:58.701828 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:58.701840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:58.702603 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:58.702611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:58.702850 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:58.703523 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:58.703546 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:58.705687 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:58.708816 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:58.708824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:58.708840 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:58.709356 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:58.709730 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:58.709735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:58.709739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:58.709920 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:58.710459 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:58.710466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:58.710793 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 11:50:58.711052 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:50:58.711576 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":266,"threads_started":1}
I20260924 11:50:58.714318 17547 heartbeater.cc:347] Connected to a master server at 127.1.210.62:41355
I20260924 11:50:58.714619 17547 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:58.715327 17547 heartbeater.cc:511] Master 127.1.210.62:41355 requested a full tablet report, sending...
I20260924 11:50:58.717641 17358 ts_manager.cc:194] Registered new tserver with Master: 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1:46141)
I20260924 11:50:58.719873 17358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:40111
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:50:58.750386 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:41355
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:59.075977 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:59.076247 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:59.076339 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:59.083461 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:59.083774 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:59.083940 17553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:50:59.093726 17553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:59.094960 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:59.096220 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:59.110540 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:59.110569 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:59.111109 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:59.111706 17553 server_base.cc:1034] running on GCE node
I20260924 11:50:59.112257 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:59.113233 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:59.114487 17553 hybrid_clock.cc:648] HybridClock initialized: now 1790250659114421 us; error 49 us; skew 500 ppm
I20260924 11:50:59.114810 17553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:59 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:59.118039 17553 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:50:59.119486 17553 webserver.cc:460] Webserver started at http://127.1.210.2:40001/ using document root <none> and password file <none>
I20260924 11:50:59.120162 17553 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:59.120296 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:59.120604 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:59.123478 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "909ec80f357c421d88995d11ffba5ca7"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "2887ba95e9949aef324a63414af01420"
server_key_iv: "74bb305650b080aaba0c9df76860300c"
server_key_version: "encryptionkey@0"
I20260924 11:50:59.124248 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "909ec80f357c421d88995d11ffba5ca7"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "2887ba95e9949aef324a63414af01420"
server_key_iv: "74bb305650b080aaba0c9df76860300c"
server_key_version: "encryptionkey@0"
I20260924 11:50:59.129405 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:50:59.132661 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:59.134083 17553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:50:59.134254 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "909ec80f357c421d88995d11ffba5ca7"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "2887ba95e9949aef324a63414af01420"
server_key_iv: "74bb305650b080aaba0c9df76860300c"
server_key_version: "encryptionkey@0"
I20260924 11:50:59.134449 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:59.150416 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:59.154322 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:59.154614 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:59.155555 17553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:59.156833 17553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:59.156937 17553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:59.157148 17553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:59.157223 17553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:59.190589 17553 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:38571
I20260924 11:50:59.190634 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:38571 every 8 connection(s)
I20260924 11:50:59.191893 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:50:59.192613 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17553
I20260924 11:50:59.192837 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:50:59.193476 1864 external_mini_cluster.cc:1428] Setting key 02ad90bfc3beb0c51860496b60da3e0a
Sep 24 11:50:59 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:59.216955 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.194780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:58757 (local address 127.1.210.62:41355)
0924 11:50:59.194971 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:59.194977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:59.196376 (+ 1399us) server_negotiation.cc:407] Connection header received
0924 11:50:59.198310 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:59.198318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:59.198408 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:59.198528 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:59.199969 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.201140 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.202655 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.203380 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.203524 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.207681 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:59.207690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:59.207698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:59.207748 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:59.211561 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:59.212747 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:59.212754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:59.212760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:59.212940 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:59.213763 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:59.213770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:59.213776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:59.214188 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:59.214337 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:59.216235 (+ 1898us) server_negotiation.cc:299] Negotiation successful
0924 11:50:59.216515 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:50:59.218075 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.195352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:58757)
0924 11:50:59.196202 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:59.196253 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:59.198056 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:59.198724 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:59.198732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:59.199237 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:59.199795 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.199812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.201285 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.201289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.202493 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.202505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.203599 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.203604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.203800 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.204478 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:59.204499 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:59.207333 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:59.211725 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:59.211737 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:59.211756 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:59.212628 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:59.213105 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:59.213113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:59.213120 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:59.213618 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:59.214408 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:59.214418 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:59.215129 (+ 711us) client_negotiation.cc:769] Sending connection context
0924 11:50:59.215440 (+ 311us) client_negotiation.cc:240] Negotiation successful
0924 11:50:59.216028 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":352,"threads_started":1}
I20260924 11:50:59.220199 17681 heartbeater.cc:347] Connected to a master server at 127.1.210.62:41355
I20260924 11:50:59.220753 17681 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:50:59.221834 17681 heartbeater.cc:511] Master 127.1.210.62:41355 requested a full tablet report, sending...
I20260924 11:50:59.223605 17358 ts_manager.cc:194] Registered new tserver with Master: 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2:38571)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
I20260924 11:50:59.224805 17358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:58757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:50:59.258396 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:41355
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:50:59.577483 17686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:50:59.577742 17686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:50:59.577955 17686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:50:59.585240 17686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:50:59.585384 17686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:50:59.585566 17686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:50:59.595152 17686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:50:59.596691 17686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:50:59.598330 17686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:50:59.612110 17692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:59.612171 17694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:50:59.612213 17691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:50:59.612691 17686 server_base.cc:1034] running on GCE node
I20260924 11:50:59.613448 17686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:50:59.614351 17686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:50:59.615577 17686 hybrid_clock.cc:648] HybridClock initialized: now 1790250659615527 us; error 66 us; skew 500 ppm
I20260924 11:50:59.615882 17686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:50:59 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:50:59.618872 17686 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:50:59.620249 17686 webserver.cc:460] Webserver started at http://127.1.210.3:35041/ using document root <none> and password file <none>
I20260924 11:50:59.620903 17686 fs_manager.cc:359] Metadata directory not provided
I20260924 11:50:59.621053 17686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:50:59.621408 17686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:50:59.624222 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "5f6854f47ab74a3d864cff9db11ab268"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "b59a22382546362134b660bf1ad71adc"
server_key_iv: "c601fb6c8737912d2ccbc3fe06e6ee49"
server_key_version: "encryptionkey@0"
I20260924 11:50:59.624969 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "5f6854f47ab74a3d864cff9db11ab268"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "b59a22382546362134b660bf1ad71adc"
server_key_iv: "c601fb6c8737912d2ccbc3fe06e6ee49"
server_key_version: "encryptionkey@0"
I20260924 11:50:59.630036 17686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:50:59.633495 17701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:59.634822 17686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:50:59.634999 17686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "5f6854f47ab74a3d864cff9db11ab268"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "b59a22382546362134b660bf1ad71adc"
server_key_iv: "c601fb6c8737912d2ccbc3fe06e6ee49"
server_key_version: "encryptionkey@0"
I20260924 11:50:59.635192 17686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:50:59.658419 17686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:50:59.662317 17686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:50:59.662647 17686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:50:59.663674 17686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:50:59.665009 17686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:50:59.665136 17686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 11:50:59.665282 17686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:50:59.665359 17686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:50:59.697994 17686 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:44525
I20260924 11:50:59.698050 17813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:44525 every 8 connection(s)
I20260924 11:50:59.699220 17686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:50:59.699651 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17686
I20260924 11:50:59.699848 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:50:59.700441 1864 external_mini_cluster.cc:1428] Setting key 9fb008120f6c1c0b1e9c4a9530fd30f6
Sep 24 11:50:59 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:50:59.719403 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.701772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:35089 (local address 127.1.210.62:41355)
0924 11:50:59.701936 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:59.701940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:59.703299 (+ 1359us) server_negotiation.cc:407] Connection header received
0924 11:50:59.705318 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:59.705323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:59.705441 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:59.705551 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:59.707018 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.707766 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.709342 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.709739 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.709877 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.713302 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:59.713306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:59.713309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:59.713341 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:59.716388 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:59.717130 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:59.717134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:59.717136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:59.717255 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:59.717702 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:59.717707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:59.717709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:59.717971 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:59.718162 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:59.718833 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 11:50:59.719050 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:50:59.720299 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.702333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:35089)
0924 11:50:59.703155 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:59.703192 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:59.705014 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:59.705782 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:59.705791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:59.706315 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:50:59.706855 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.706872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.707910 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.707923 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.709195 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.709208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.709960 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.709970 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.710214 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.710873 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:50:59.710894 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:50:59.713036 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:50:59.716525 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:59.716541 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:59.716556 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:59.716961 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:59.717367 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:50:59.717373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:50:59.717377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:50:59.717562 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:50:59.718153 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:50:59.718160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:50:59.718470 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 11:50:59.718742 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 11:50:59.719250 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":356,"threads_started":1}
I20260924 11:50:59.721554 17814 heartbeater.cc:347] Connected to a master server at 127.1.210.62:41355
I20260924 11:50:59.721853 17814 heartbeater.cc:464] Registering TS with master...
I20260924 11:50:59.722529 17814 heartbeater.cc:511] Master 127.1.210.62:41355 requested a full tablet report, sending...
I20260924 11:50:59.722955 17547 heartbeater.cc:503] Master 127.1.210.62:41355 was elected leader, sending a full tablet report...
I20260924 11:50:59.724406 17357 ts_manager.cc:194] Registered new tserver with Master: 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525)
I20260924 11:50:59.725380 17357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:35089
I20260924 11:50:59.728277 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:50:59.744419 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.732523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.1.210.62:41355)
0924 11:50:59.732736 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:59.732745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:59.733124 (+ 379us) server_negotiation.cc:407] Connection header received
0924 11:50:59.733294 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:59.733298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:59.733354 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:59.733431 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:50:59.734786 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.735484 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.736704 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.737155 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.737293 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.739652 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:50:59.739658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:50:59.739662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:50:59.739697 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:50:59.742188 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:59.742784 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:59.742787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:59.742790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:59.742910 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:50:59.743216 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:50:59.743219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:50:59.743221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:50:59.743483 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:50:59.743688 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:59.743949 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 11:50:59.744121 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:50:59.748224 17357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:50:59.755191 17357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:50:59.789206 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.767613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:44525 (local address 127.0.0.1:47958)
0924 11:50:59.778074 (+ 10461us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:59.778106 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:59.778259 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:59.781772 (+ 3513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:59.781777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:59.781800 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:59.782191 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.782203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.784373 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.784379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.786383 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.786395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.788129 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.788136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.788328 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.788375 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 11:50:59.788490 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 11:50:59.788682 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10250,"mutex_wait_us":105,"thread_start_us":10098,"threads_started":1}
I20260924 11:50:59.789892 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.768197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.1.210.3:44525)
0924 11:50:59.781174 (+ 12977us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:59.781186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:59.781218 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:50:59.781316 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:59.781322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:59.781551 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:59.782378 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:59.782661 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.784215 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.786559 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.787901 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.788040 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.789296 (+ 1256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:59.789388 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 11:50:59.789565 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12762,"thread_start_us":13443,"threads_started":1}
I20260924 11:50:59.789884 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.767013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:38571 (local address 127.0.0.1:43270)
0924 11:50:59.767744 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:59.767799 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:59.767972 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:59.781344 (+ 13372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:59.781351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:59.781381 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:59.781783 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.781795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.784086 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.784091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.786872 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.786886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.788932 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.788938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.789215 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.789248 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:50:59.789341 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 11:50:59.789465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":5,"thread_start_us":563,"threads_started":1}
I20260924 11:50:59.790540 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.767232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.1.210.2:38571)
0924 11:50:59.780619 (+ 13387us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:59.780628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:59.780655 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:50:59.780771 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:59.780781 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:59.781082 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:59.782021 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:59.782256 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.783910 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.787049 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.788479 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.788602 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.788788 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:59.790156 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0924 11:50:59.790294 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13215,"thread_start_us":13240,"threads_started":1}
I20260924 11:50:59.791765 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.768484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38544)
0924 11:50:59.780278 (+ 11794us) negotiation.cc:107] Waiting for socket to connect
0924 11:50:59.780303 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:50:59.780417 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:50:59.784706 (+ 4289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:50:59.784711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:50:59.784736 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:50:59.785306 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.785320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.787340 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.787345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.789160 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:50:59.789179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.790947 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.790953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:50:59.791135 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.791172 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:50:59.791277 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 11:50:59.791401 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11667,"mutex_wait_us":33,"thread_start_us":11525,"threads_started":1}
I20260924 11:50:59.794122 17749 tablet_service.cc:1467] Processing CreateTablet for tablet c237dba3f3b84957b201f92d9b7025a1 (DEFAULT_TABLE table=test-table [id=5ffa4f7f86404f55a8857b5f724d01a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:59.794100 17616 tablet_service.cc:1467] Processing CreateTablet for tablet c237dba3f3b84957b201f92d9b7025a1 (DEFAULT_TABLE table=test-table [id=5ffa4f7f86404f55a8857b5f724d01a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:59.795280 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c237dba3f3b84957b201f92d9b7025a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:59.795292 17749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c237dba3f3b84957b201f92d9b7025a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:59.799574 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:50:59.768861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.1.210.1:46141)
0924 11:50:59.783905 (+ 15044us) server_negotiation.cc:206] Beginning negotiation
0924 11:50:59.783912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:50:59.783933 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:50:59.784067 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:50:59.784075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:50:59.784304 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:50:59.785537 (+ 1233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:50:59.785802 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.787208 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.789365 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:50:59.790745 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:50:59.790900 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:50:59.791126 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 11:50:59.791333 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 11:50:59.791497 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14912,"thread_start_us":15584,"threads_started":1}
I20260924 11:50:59.803395 17482 tablet_service.cc:1467] Processing CreateTablet for tablet c237dba3f3b84957b201f92d9b7025a1 (DEFAULT_TABLE table=test-table [id=5ffa4f7f86404f55a8857b5f724d01a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:50:59.804811 17482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c237dba3f3b84957b201f92d9b7025a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:50:59.807766 17833 tablet_bootstrap.cc:492] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Bootstrap starting.
I20260924 11:50:59.809476 17832 tablet_bootstrap.cc:492] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Bootstrap starting.
I20260924 11:50:59.811296 17833 tablet_bootstrap.cc:654] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:59.812278 17833 log.cc:824] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:59.812857 17832 tablet_bootstrap.cc:654] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:59.814111 17832 log.cc:824] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:59.815085 17833 tablet_bootstrap.cc:492] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: No bootstrap required, opened a new log
I20260924 11:50:59.815397 17833 ts_tablet_manager.cc:1397] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 11:50:59.817004 17832 tablet_bootstrap.cc:492] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: No bootstrap required, opened a new log
I20260924 11:50:59.817270 17832 ts_tablet_manager.cc:1397] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260924 11:50:59.819944 17836 tablet_bootstrap.cc:492] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Bootstrap starting.
I20260924 11:50:59.822921 17836 tablet_bootstrap.cc:654] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Neither blocks nor log segments found. Creating new log.
I20260924 11:50:59.824301 17836 log.cc:824] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Log is configured to *not* fsync() on all Append() calls
I20260924 11:50:59.827579 17836 tablet_bootstrap.cc:492] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: No bootstrap required, opened a new log
I20260924 11:50:59.827848 17836 ts_tablet_manager.cc:1397] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 11:50:59.827682 17833 raft_consensus.cc:348] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:50:59.828153 17833 raft_consensus.cc:374] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:59.828258 17833 raft_consensus.cc:729] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909ec80f357c421d88995d11ffba5ca7, State: Initialized, Role: FOLLOWER
I20260924 11:50:59.828121 17832 raft_consensus.cc:348] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:50:59.828552 17832 raft_consensus.cc:374] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:59.828645 17832 raft_consensus.cc:729] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6854f47ab74a3d864cff9db11ab268, State: Initialized, Role: FOLLOWER
I20260924 11:50:59.828879 17833 consensus_queue.cc:260] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:50:59.829385 17832 consensus_queue.cc:260] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:50:59.830232 17681 heartbeater.cc:503] Master 127.1.210.62:41355 was elected leader, sending a full tablet report...
I20260924 11:50:59.832582 17833 ts_tablet_manager.cc:1428] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260924 11:50:59.833572 17814 heartbeater.cc:503] Master 127.1.210.62:41355 was elected leader, sending a full tablet report...
I20260924 11:50:59.834242 17832 ts_tablet_manager.cc:1428] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260924 11:50:59.838250 17836 raft_consensus.cc:348] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:50:59.838615 17836 raft_consensus.cc:374] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:50:59.838735 17836 raft_consensus.cc:729] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997973b2f58f4be2ad3308a17cc6c953, State: Initialized, Role: FOLLOWER
I20260924 11:50:59.839347 17836 consensus_queue.cc:260] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:50:59.841032 17836 ts_tablet_manager.cc:1428] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260924 11:50:59.944002 17682 tablet.cc:2369] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:50:59.944413 17548 tablet.cc:2369] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:50:59.950928 17815 tablet.cc:2369] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:51:00.033013 17838 raft_consensus.cc:482] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:51:00.033318 17838 raft_consensus.cc:504] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:51:00.035005 17838 leader_election.cc:288] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1:46141), 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525)
I20260924 11:51:00.042728 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.035516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57501 (local address 127.1.210.1:46141)
0924 11:51:00.035739 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:00.035745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:00.036614 (+ 869us) server_negotiation.cc:407] Connection header received
0924 11:51:00.036781 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:00.036787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:00.036866 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:00.036965 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:00.037726 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.038570 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.040821 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.041727 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.041832 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.041908 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:00.042330 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 11:51:00.042478 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:51:00.042845 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.035657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:58169 (local address 127.1.210.3:44525)
0924 11:51:00.035899 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:00.035908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:00.036800 (+ 892us) server_negotiation.cc:407] Connection header received
0924 11:51:00.036995 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:00.037001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:00.037111 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:00.037213 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:00.037921 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.038957 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.041109 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.041963 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.042090 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.042185 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:00.042476 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 11:51:00.042613 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:51:00.042833 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.035390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.1.210.2:57501)
0924 11:51:00.036484 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.036514 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:00.036638 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:00.036987 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:00.036992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:00.037017 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:00.037510 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:00.037561 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.038704 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.038710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:00.040654 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:00.040666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.041972 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.041978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:00.042148 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.042183 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:51:00.042278 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:51:00.042370 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":67,"thread_start_us":656,"threads_started":1}
I20260924 11:51:00.043373 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.035662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:44525 (local address 127.1.210.2:58169)
0924 11:51:00.036656 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.036675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:00.036822 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:00.037244 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:00.037247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:00.037268 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:00.037726 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:00.037737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.039130 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.039136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:00.040901 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:00.040914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.042157 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.042161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:00.042313 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.042344 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:51:00.042431 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:51:00.042552 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":21,"thread_start_us":593,"threads_started":1}
I20260924 11:51:00.044003 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c237dba3f3b84957b201f92d9b7025a1" candidate_uuid: "909ec80f357c421d88995d11ffba5ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997973b2f58f4be2ad3308a17cc6c953" is_pre_election: true
I20260924 11:51:00.044438 17769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c237dba3f3b84957b201f92d9b7025a1" candidate_uuid: "909ec80f357c421d88995d11ffba5ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6854f47ab74a3d864cff9db11ab268" is_pre_election: true
I20260924 11:51:00.044642 17502 raft_consensus.cc:2454] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909ec80f357c421d88995d11ffba5ca7 in term 0.
I20260924 11:51:00.044968 17769 raft_consensus.cc:2454] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909ec80f357c421d88995d11ffba5ca7 in term 0.
I20260924 11:51:00.045482 17571 leader_election.cc:302] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909ec80f357c421d88995d11ffba5ca7, 997973b2f58f4be2ad3308a17cc6c953; no voters:
I20260924 11:51:00.045951 17838 raft_consensus.cc:2781] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:51:00.046114 17838 raft_consensus.cc:482] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:51:00.046226 17838 raft_consensus.cc:3037] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:51:00.048283 17838 raft_consensus.cc:504] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:51:00.048898 17838 leader_election.cc:288] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [CANDIDATE]: Term 1 election: Requested vote from peers 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1:46141), 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525)
I20260924 11:51:00.049257 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c237dba3f3b84957b201f92d9b7025a1" candidate_uuid: "909ec80f357c421d88995d11ffba5ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997973b2f58f4be2ad3308a17cc6c953"
I20260924 11:51:00.049360 17769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c237dba3f3b84957b201f92d9b7025a1" candidate_uuid: "909ec80f357c421d88995d11ffba5ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6854f47ab74a3d864cff9db11ab268"
I20260924 11:51:00.049477 17502 raft_consensus.cc:3037] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:51:00.049619 17769 raft_consensus.cc:3037] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:51:00.051797 17769 raft_consensus.cc:2454] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909ec80f357c421d88995d11ffba5ca7 in term 1.
I20260924 11:51:00.051797 17502 raft_consensus.cc:2454] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909ec80f357c421d88995d11ffba5ca7 in term 1.
I20260924 11:51:00.052224 17571 leader_election.cc:302] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909ec80f357c421d88995d11ffba5ca7, 997973b2f58f4be2ad3308a17cc6c953; no voters:
I20260924 11:51:00.052548 17838 raft_consensus.cc:2781] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:51:00.053032 17838 raft_consensus.cc:686] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 1 LEADER]: Becoming Leader. State: Replica: 909ec80f357c421d88995d11ffba5ca7, State: Running, Role: LEADER
I20260924 11:51:00.053598 17838 consensus_queue.cc:237] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } }
I20260924 11:51:00.057955 17358 catalog_manager.cc:5495] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "909ec80f357c421d88995d11ffba5ca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } health_report { overall_health: UNKNOWN } } }
I20260924 11:51:00.079958 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.075212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.1.210.2:38571)
0924 11:51:00.075384 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:00.075388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:00.075422 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:51:00.075605 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:00.075608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:00.075682 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:00.075800 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:51:00.076124 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.076816 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.078223 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.078593 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.078685 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.079077 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:51:00.079517 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:51:00.079624 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:00.079702 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 11:51:00.079760 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:51:00.087862 17769 raft_consensus.cc:1261] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Refusing update from remote peer 909ec80f357c421d88995d11ffba5ca7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:51:00.087862 17502 raft_consensus.cc:1261] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Refusing update from remote peer 909ec80f357c421d88995d11ffba5ca7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:51:00.088886 17838 consensus_queue.cc:1035] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:51:00.089167 17843 consensus_queue.cc:1035] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:51:00.100278 17846 mvcc.cc:205] Tried to move back new op lower bound from 7332866703709978624 to 7332866703583547392. Current Snapshot: MvccSnapshot[applied={T|T < 7332866703709978624}]
I20260924 11:51:00.128033 17357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:51:00.128353 17357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:51:00.133080 17357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Sending DeleteTablet for 3 replicas of tablet c237dba3f3b84957b201f92d9b7025a1
I20260924 11:51:00.134295 17749 tablet_service.cc:1514] Processing DeleteTablet for tablet c237dba3f3b84957b201f92d9b7025a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:00 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:47958
I20260924 11:51:00.134493 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17419
I20260924 11:51:00.137102 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet c237dba3f3b84957b201f92d9b7025a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:00 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260924 11:51:00.138062 17482 tablet_service.cc:1514] Processing DeleteTablet for tablet c237dba3f3b84957b201f92d9b7025a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:00 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:38544
I20260924 11:51:00.138121 17481 tablet_service.cc:1514] Processing DeleteTablet for tablet c237dba3f3b84957b201f92d9b7025a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:00 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:38544
I20260924 11:51:00.146950 17859 tablet_replica.cc:330] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: stopping tablet replica
I20260924 11:51:00.148394 17856 tablet_replica.cc:330] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: stopping tablet replica
I20260924 11:51:00.148700 17858 tablet_replica.cc:330] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: stopping tablet replica
I20260924 11:51:00.148847 17856 raft_consensus.cc:2229] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:51:00.149171 17858 raft_consensus.cc:2229] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:51:00.149485 17856 raft_consensus.cc:2258] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:51:00.149940 17858 raft_consensus.cc:2258] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:51:00.151878 17856 ts_tablet_manager.cc:1905] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:00.152420 17858 ts_tablet_manager.cc:1905] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:00.147507 17859 raft_consensus.cc:2229] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 24 11:51:00 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:51:00.159260 17856 ts_tablet_manager.cc:1918] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:51:00.159428 17856 log.cc:1196] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/c237dba3f3b84957b201f92d9b7025a1
I20260924 11:51:00.159967 17856 ts_tablet_manager.cc:1939] T c237dba3f3b84957b201f92d9b7025a1 P 5f6854f47ab74a3d864cff9db11ab268: Deleting consensus metadata
I20260924 11:51:00.160157 17858 ts_tablet_manager.cc:1918] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:51:00.160338 17858 log.cc:1196] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/c237dba3f3b84957b201f92d9b7025a1
I20260924 11:51:00.160852 17858 ts_tablet_manager.cc:1939] T c237dba3f3b84957b201f92d9b7025a1 P 909ec80f357c421d88995d11ffba5ca7: Deleting consensus metadata
I20260924 11:51:00.161947 17342 catalog_manager.cc:4841] TS 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525): tablet c237dba3f3b84957b201f92d9b7025a1 (table test-table [id=5ffa4f7f86404f55a8857b5f724d01a4]) successfully deleted
I20260924 11:51:00.163129 17859 raft_consensus.cc:2258] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:51:00.163162 17344 catalog_manager.cc:4841] TS 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2:38571): tablet c237dba3f3b84957b201f92d9b7025a1 (table test-table [id=5ffa4f7f86404f55a8857b5f724d01a4]) successfully deleted
I20260924 11:51:00.165542 17859 ts_tablet_manager.cc:1905] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:00.167821 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.135771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.1.210.1:46141)
0924 11:51:00.136002 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:00.136011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:00.147595 (+ 11584us) server_negotiation.cc:407] Connection header received
0924 11:51:00.147857 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:00.147865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:00.148010 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:00.148137 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:00.150655 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.151823 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.153858 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.154508 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.154612 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.160111 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:00.160129 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:00.160150 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:00.160212 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:00.164282 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:00.165203 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:00.165211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:00.165215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:00.165381 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:00.165834 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:00.165839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:00.165842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:00.166275 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:00.166555 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:00.167176 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 11:51:00.167407 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:51:00.169306 17542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:00.290365 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17419
W20260924 11:51:00.313301 17344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.1:46141 (error 108)
I20260924 11:51:00.314626 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17553
I20260924 11:51:00.315482 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.314509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38560)
0924 11:51:00.315153 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.315278 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":393,"threads_started":1}
W20260924 11:51:00.315855 17344 catalog_manager.cc:4584] TS 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1:46141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c237dba3f3b84957b201f92d9b7025a1: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Sep 24 11:51:00 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:51:00.332176 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.315417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.1.210.2:38571)
0924 11:51:00.315614 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:00.315623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:00.315709 (+ 86us) server_negotiation.cc:407] Connection header received
0924 11:51:00.315931 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:00.315937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:00.316047 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:00.316143 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:00.317703 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.318917 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.320315 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.320971 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.321112 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.325314 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:00.325330 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:00.325350 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:00.325410 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:00.329147 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:00.329874 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:00.329890 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:00.329893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:00.330034 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:00.330481 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:00.330487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:00.330530 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:00.330968 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:00.331291 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:00.331571 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 11:51:00.331794 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:51:00.332930 17676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:00.360471 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.359951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38574)
0924 11:51:00.360218 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.360313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 11:51:00.361572 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.361214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38578)
0924 11:51:00.361356 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.361439 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:51:00.369686 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.369379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38586)
0924 11:51:00.369532 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.369590 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:51:00.370540 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.370234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38590)
0924 11:51:00.370363 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.370424 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:51:00.409857 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.409402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38602)
0924 11:51:00.409623 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.409706 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 11:51:00.410897 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.410502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38608)
0924 11:51:00.410716 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.410778 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 11:51:00.447155 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17553
I20260924 11:51:00.448473 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.446884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38610)
0924 11:51:00.447092 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.447200 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 11:51:00.449589 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.449199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38614)
0924 11:51:00.449340 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.449412 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 11:51:00.465785 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17686
Sep 24 11:51:00 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:51:00.480858 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.466518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.1.210.3:44525)
0924 11:51:00.466708 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:00.466715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:00.466735 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:51:00.466912 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:00.466924 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:00.466989 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:00.467098 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:00.468489 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.469277 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.470612 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:00.471002 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:00.471139 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:00.475537 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:00.475551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:00.475567 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:00.475604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:00.478110 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:00.478802 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:00.478810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:00.478812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:00.478951 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:00.479309 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:00.479315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:00.479318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:00.479678 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:00.479967 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:00.480270 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:51:00.480478 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:51:00.481547 17809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:00.511916 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.511391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38624)
0924 11:51:00.511621 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.511734 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:51:00.513128 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.512684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38630)
0924 11:51:00.512853 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.512937 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 11:51:00.551548 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.551000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38644)
0924 11:51:00.551251 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.551360 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 11:51:00.552613 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:00.552177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38648)
0924 11:51:00.552357 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:00.552460 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:51:00.571327 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17686
I20260924 11:51:00.587821 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17326
I20260924 11:51:00.588925 17388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:00.698697 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17326
I20260924 11:51:20.720110 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41355
--webserver_interface=127.1.210.62
--webserver_port=43837
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:51:21.048445 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:21.048727 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:21.048884 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:21.056033 17866 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:51:21.056120 17866 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:51:21.056217 17866 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:51:21.056303 17866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:51:21.056373 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:21.056486 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:21.056563 17866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:51:21.056648 17866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:51:21.066252 17866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=43837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:21.067474 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:21.068811 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:21.091333 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:21.091295 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:21.091409 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:21.091571 17866 server_base.cc:1034] running on GCE node
I20260924 11:51:21.092650 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:21.094272 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:21.095492 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790250681095436 us; error 37 us; skew 500 ppm
I20260924 11:51:21.095893 17866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:21 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250681, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:21.099501 17866 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:51:21.101012 17866 webserver.cc:460] Webserver started at http://127.1.210.62:43837/ using document root <none> and password file <none>
I20260924 11:51:21.101823 17866 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:21.101925 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:21.107868 17866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 11:51:21.110762 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:21.112123 17866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:51:21.112329 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "2667fc4a393c48529c9c3ad031a6fe0e"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "a9336c68f37225810e30eec8f586e28c"
server_key_iv: "25cdb024bf0c70421928bf9f2a28e223"
server_key_version: "encryptionkey@0"
I20260924 11:51:21.113451 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:21.134159 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:21.138363 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:21.138662 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:21.156579 17866 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:41355
I20260924 11:51:21.156607 17933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:41355 every 8 connection(s)
I20260924 11:51:21.158154 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:51:21.163028 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17866
I20260924 11:51:21.163861 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:46141
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:41355
--webserver_port=39641
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:51:21.165499 17934 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:51:21.168745 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap starting.
I20260924 11:51:21.197710 17934 log.cc:824] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:51:21.213442 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:51:21.214133 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap complete.
I20260924 11:51:21.225752 17934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:21.226612 17934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2667fc4a393c48529c9c3ad031a6fe0e, State: Initialized, Role: FOLLOWER
I20260924 11:51:21.227214 17934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:21.227464 17934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:51:21.227596 17934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:51:21.227777 17934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:51:21.231707 17934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:21.232220 17934 leader_election.cc:302] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2667fc4a393c48529c9c3ad031a6fe0e; no voters:
I20260924 11:51:21.232807 17934 leader_election.cc:288] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:51:21.233008 17938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:51:21.234318 17938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 LEADER]: Becoming Leader. State: Replica: 2667fc4a393c48529c9c3ad031a6fe0e, State: Running, Role: LEADER
I20260924 11:51:21.234879 17938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:21.235085 17934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:51:21.237882 17939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } } }
I20260924 11:51:21.238245 17939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: This master's current role is: LEADER
I20260924 11:51:21.237461 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2667fc4a393c48529c9c3ad031a6fe0e. Latest consensus state: current_term: 2 leader_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } } }
I20260924 11:51:21.238878 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: This master's current role is: LEADER
I20260924 11:51:21.249737 17945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:51:21.259635 17945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:51:21.261445 17945 catalog_manager.cc:1259] Loaded cluster ID: 98619ab58ba742f59a5091981907a393
I20260924 11:51:21.261564 17945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:51:21.266588 17945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:51:21.268517 17945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Loaded TSK: 0
I20260924 11:51:21.320971 17945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Generated new TSK 1
I20260924 11:51:21.321527 17945 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:51:21.562832 17936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:21.563133 17936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:21.563300 17936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:21.570773 17936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:21.570930 17936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:21.571100 17936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:51:21.580760 17936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:46141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=39641
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:21.582121 17936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:21.583473 17936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:21.598245 17962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:21.598229 17964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:21.598775 17961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:21.599226 17936 server_base.cc:1034] running on GCE node
I20260924 11:51:21.599895 17936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:21.600937 17936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:21.602167 17936 hybrid_clock.cc:648] HybridClock initialized: now 1790250681602103 us; error 63 us; skew 500 ppm
I20260924 11:51:21.602487 17936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:21 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250681, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:21.605629 17936 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:51:21.607155 17936 webserver.cc:460] Webserver started at http://127.1.210.1:39641/ using document root <none> and password file <none>
I20260924 11:51:21.607817 17936 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:21.607955 17936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:21.613435 17936 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:51:21.616385 17971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:21.617859 17936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:51:21.618012 17936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "997973b2f58f4be2ad3308a17cc6c953"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "dd7172f1cf17d474e194dc012a55d777"
server_key_iv: "4ac01cb3d0f39912ce8f8a2cebbf8f8d"
server_key_version: "encryptionkey@0"
I20260924 11:51:21.619040 17936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:21.645460 17936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:21.649945 17936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:21.650229 17936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:21.651355 17936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:51:21.653409 17978 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 11:51:21.658679 17978 ts_tablet_manager.cc:1852] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 11:51:21.659644 17978 ts_tablet_manager.cc:1905] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:21.666828 17978 ts_tablet_manager.cc:1918] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:51:21.667035 17978 log.cc:1196] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/c237dba3f3b84957b201f92d9b7025a1
I20260924 11:51:21.667524 17978 ts_tablet_manager.cc:1939] T c237dba3f3b84957b201f92d9b7025a1 P 997973b2f58f4be2ad3308a17cc6c953: Deleting consensus metadata
I20260924 11:51:21.667950 17936 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 11:51:21.668066 17936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260924 11:51:21.668212 17936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:51:21.668303 17936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:21.705219 17936 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:46141
I20260924 11:51:21.705247 18084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:46141 every 8 connection(s)
I20260924 11:51:21.706576 17936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:51:21.712337 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 17936
I20260924 11:51:21.713150 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:38571
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:41355
--webserver_port=40001
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:51:21 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250681, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:51:21.732811 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:21.709692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:51021)
0924 11:51:21.710454 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:21.710495 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:21.712219 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:21.713827 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:21.713835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:21.714346 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:51:21.714876 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:21.714893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:21.717352 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:21.717364 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:21.718695 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:21.718707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:21.720128 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:21.720133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:21.720321 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:21.721099 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:51:21.721139 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:51:21.723986 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:51:21.728663 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:51:21.728670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:51:21.728681 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:51:21.729267 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:51:21.729759 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:51:21.729763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:51:21.729765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:51:21.729927 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:51:21.730924 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:51:21.730930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:51:21.731228 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 11:51:21.731459 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 11:51:21.731850 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":305,"threads_started":1}
I20260924 11:51:21.733954 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:21.709391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:51021 (local address 127.1.210.62:41355)
0924 11:51:21.710007 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:21.710027 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:21.710623 (+ 596us) server_negotiation.cc:407] Connection header received
0924 11:51:21.712815 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:21.712836 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:21.713337 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:21.713725 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:21.715122 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:21.717140 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:21.718826 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:21.719916 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:21.720109 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:21.724253 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:21.724265 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:21.724287 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:21.724337 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:21.728501 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:21.729402 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:21.729413 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:21.729424 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:21.729641 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:21.730051 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:21.730058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:21.730064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:21.730757 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:21.730948 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:21.731749 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 11:51:21.732193 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":263,"threads_started":1}
I20260924 11:51:21.736892 18085 heartbeater.cc:347] Connected to a master server at 127.1.210.62:41355
I20260924 11:51:21.737267 18085 heartbeater.cc:464] Registering TS with master...
I20260924 11:51:21.738178 18085 heartbeater.cc:511] Master 127.1.210.62:41355 requested a full tablet report, sending...
I20260924 11:51:21.740844 17899 ts_manager.cc:194] Registered new tserver with Master: 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1:46141)
I20260924 11:51:21.743046 17899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:51021
W20260924 11:51:22.049039 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:22.049350 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:22.049443 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:22.056690 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:22.056800 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:22.056910 18089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:51:22.066861 18089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:38571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=40001
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:22.068040 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:22.069367 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:22.082129 18094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:22.082100 18097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:22.082237 18095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:22.083250 18089 server_base.cc:1034] running on GCE node
I20260924 11:51:22.083895 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:22.084867 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:22.086097 18089 hybrid_clock.cc:648] HybridClock initialized: now 1790250682086051 us; error 38 us; skew 500 ppm
I20260924 11:51:22.086498 18089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:22 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250682, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:22.089592 18089 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:51:22.091030 18089 webserver.cc:460] Webserver started at http://127.1.210.2:40001/ using document root <none> and password file <none>
I20260924 11:51:22.091710 18089 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:22.091864 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:22.097008 18089 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:51:22.099674 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:22.100951 18089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:51:22.101162 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "909ec80f357c421d88995d11ffba5ca7"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "2887ba95e9949aef324a63414af01420"
server_key_iv: "74bb305650b080aaba0c9df76860300c"
server_key_version: "encryptionkey@0"
I20260924 11:51:22.102098 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:22.124243 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:22.128165 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:22.128438 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:22.129472 18089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:51:22.130861 18089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:51:22.130980 18089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:22.131115 18089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:51:22.131208 18089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:22.164831 18089 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:38571
I20260924 11:51:22.164880 18216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:38571 every 8 connection(s)
I20260924 11:51:22.166335 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:51:22.170827 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18089
I20260924 11:51:22.171511 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:44525
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:41355
--webserver_port=35041
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:51:22 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250682, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:51:22.187997 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.169041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:43519 (local address 127.1.210.62:41355)
0924 11:51:22.169264 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.169270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.170573 (+ 1303us) server_negotiation.cc:407] Connection header received
0924 11:51:22.173211 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.173220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.173357 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.173476 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:22.175315 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.176467 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.178201 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.178695 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.178840 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.182382 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:22.182387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:22.182390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:22.182423 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:22.184978 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.185743 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.185748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.185753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.185875 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.186307 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.186312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.186316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.186584 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:22.186748 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.187419 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.187652 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 11:51:22.188970 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.169593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:43519)
0924 11:51:22.170368 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.170461 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.172881 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.173750 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.173760 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.174429 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:51:22.175132 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.175149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.176618 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.176621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.178041 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.178054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.178933 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.178940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.179172 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.179887 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:51:22.179910 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:51:22.182140 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:51:22.185137 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:51:22.185147 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:51:22.185165 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:51:22.185607 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:51:22.185988 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:51:22.185993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:51:22.185997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:51:22.186189 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:51:22.186718 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:51:22.186726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:51:22.187065 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.187329 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.187876 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":304,"threads_started":1}
I20260924 11:51:22.190665 18217 heartbeater.cc:347] Connected to a master server at 127.1.210.62:41355
I20260924 11:51:22.190975 18217 heartbeater.cc:464] Registering TS with master...
I20260924 11:51:22.191681 18217 heartbeater.cc:511] Master 127.1.210.62:41355 requested a full tablet report, sending...
I20260924 11:51:22.193188 17899 ts_manager.cc:194] Registered new tserver with Master: 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2:38571)
I20260924 11:51:22.194010 17899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:43519
W20260924 11:51:22.495847 18220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:22.496161 18220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:22.496338 18220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:22.503616 18220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:22.503782 18220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:22.503959 18220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:51:22.513790 18220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:44525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=35041
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:22.515094 18220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:22.516405 18220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:22.530436 18228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:22.530694 18225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:22.530691 18226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:22.531806 18220 server_base.cc:1034] running on GCE node
I20260924 11:51:22.532433 18220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:22.533398 18220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:22.534610 18220 hybrid_clock.cc:648] HybridClock initialized: now 1790250682534569 us; error 40 us; skew 500 ppm
I20260924 11:51:22.534924 18220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:22 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250682, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:22.538270 18220 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:51:22.539740 18220 webserver.cc:460] Webserver started at http://127.1.210.3:35041/ using document root <none> and password file <none>
I20260924 11:51:22.540412 18220 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:22.540566 18220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:22.545917 18220 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:51:22.548401 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:22.549728 18220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:51:22.549906 18220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "5f6854f47ab74a3d864cff9db11ab268"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "b59a22382546362134b660bf1ad71adc"
server_key_iv: "c601fb6c8737912d2ccbc3fe06e6ee49"
server_key_version: "encryptionkey@0"
I20260924 11:51:22.550866 18220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:22.589591 18220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:22.593820 18220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:22.594090 18220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:22.595232 18220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:51:22.596549 18220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:51:22.596640 18220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:22.596796 18220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:51:22.596894 18220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:22.629851 18220 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:44525
I20260924 11:51:22.629942 18347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:44525 every 8 connection(s)
I20260924 11:51:22.631512 18220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:51:22.635776 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18220
Sep 24 11:51:22 dist-test-slave-dd32 krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250682, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:51:22.652055 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.636598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.1.210.62:41355)
0924 11:51:22.637086 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.637093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.637507 (+ 414us) server_negotiation.cc:407] Connection header received
0924 11:51:22.637640 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.637647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.637736 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.637873 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:22.639578 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.640431 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.641805 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.642308 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.642473 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.645537 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:22.645542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:22.645550 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:22.645580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:22.649207 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.649904 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.649908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.649910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.650027 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.650429 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.650432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.650434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.650820 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:22.650987 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.651454 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.651680 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260924 11:51:22.653190 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.634163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:38297 (local address 127.1.210.62:41355)
0924 11:51:22.634346 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.634350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.635469 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 11:51:22.637677 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.637681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.637743 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.637814 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:22.639323 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.640173 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.641845 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.642310 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.642502 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.646155 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:22.646159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:22.646162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:22.646189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:22.649714 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.650563 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.650568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.650570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.650679 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.651324 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.651329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.651337 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.651680 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:22.651832 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.652602 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.652811 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":8064}
I20260924 11:51:22.654389 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.634603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:38297)
0924 11:51:22.635309 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.635345 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.637389 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.638105 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.638120 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.638609 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:51:22.639149 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.639166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.640338 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.640343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.641680 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.641692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.642531 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.642540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.642794 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.643547 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:51:22.643568 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:51:22.645957 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:51:22.649845 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:51:22.649855 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:51:22.649871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:51:22.650449 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:51:22.650778 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:51:22.650781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:51:22.650784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:51:22.651018 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:51:22.651830 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:51:22.651836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:51:22.652180 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.652487 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.653007 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":272,"threads_started":1}
I20260924 11:51:22.655761 18348 heartbeater.cc:347] Connected to a master server at 127.1.210.62:41355
I20260924 11:51:22.656107 18348 heartbeater.cc:464] Registering TS with master...
I20260924 11:51:22.656796 18348 heartbeater.cc:511] Master 127.1.210.62:41355 requested a full tablet report, sending...
I20260924 11:51:22.658097 17899 ts_manager.cc:194] Registered new tserver with Master: 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525)
I20260924 11:51:22.658877 17899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:38297
I20260924 11:51:22.666778 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:51:22.672811 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.667621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.1.210.62:41355)
0924 11:51:22.667830 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.667839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.668265 (+ 426us) server_negotiation.cc:407] Connection header received
0924 11:51:22.668429 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.668433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.668558 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.668667 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:51:22.669123 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.669849 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.671254 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.671646 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.671734 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.672147 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:51:22.672332 (+ 185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 11:51:22.672591 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39480: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:51:22.672932 18087 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39480: token expired
W20260924 11:51:22.673470 17818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.210.62:41355)
I20260924 11:51:22.685958 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.673866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.1.210.62:41355)
0924 11:51:22.674022 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.674026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.674040 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:51:22.674151 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.674155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.674205 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.674285 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:22.675502 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.676183 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.677650 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.678032 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.678122 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.680576 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:22.680581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:22.680585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:22.680618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:22.683532 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.684141 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.684146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.684148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.684275 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.684614 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.684618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.684620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.684935 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:22.685226 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.685424 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.685613 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:51:22.689373 17899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:51:22.696522 17899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:51:22.735586 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.709329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.1.210.1:46141)
0924 11:51:22.723398 (+ 14069us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.723411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.724773 (+ 1362us) server_negotiation.cc:407] Connection header received
0924 11:51:22.725145 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.725153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.725445 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.726883 (+ 1438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:22.728048 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.729589 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.732563 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.734084 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.734207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.734423 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.735033 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.735205 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13839,"thread_start_us":13675,"threads_started":1}
I20260924 11:51:22.735572 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.708682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46141 (local address 127.0.0.1:38112)
0924 11:51:22.724618 (+ 15936us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.724659 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.724861 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.727193 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.727201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.727260 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:22.727881 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.727893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.729731 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.729736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.732370 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.732385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.734621 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.734629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.734842 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.734873 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.734978 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.735128 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15708,"mutex_wait_us":83,"thread_start_us":15615,"threads_started":1}
I20260924 11:51:22.735960 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.709422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.1.210.3:44525)
0924 11:51:22.724264 (+ 14842us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.724274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.725859 (+ 1585us) server_negotiation.cc:407] Connection header received
0924 11:51:22.725984 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.725994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.726335 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.726565 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:22.728499 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.730000 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.732536 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.733840 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.733985 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.734250 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.735235 (+ 985us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.735416 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14623,"thread_start_us":14368,"threads_started":1}
I20260924 11:51:22.736598 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.708878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:44525 (local address 127.0.0.1:60620)
0924 11:51:22.725598 (+ 16720us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.725616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.725735 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.727189 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.727201 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.727753 (+ 552us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:22.728282 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.728292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.730157 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.730162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.732358 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.732373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.734398 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.734404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.734612 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.734677 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.734881 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.735081 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16578,"mutex_wait_us":98,"spinlock_wait_cycles":1107200,"thread_start_us":18004,"threads_started":1}
I20260924 11:51:22.736626 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.725410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:38571 (local address 127.0.0.1:46550)
0924 11:51:22.728138 (+ 2728us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.728159 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.728293 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.728879 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.728884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.728911 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:22.729256 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.729267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.731024 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.731030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.733652 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.733667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.735767 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.735773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.735934 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.735967 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.736055 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.736200 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2578,"mutex_wait_us":71,"thread_start_us":2247,"threads_started":1}
I20260924 11:51:22.737160 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.725681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.1.210.2:38571)
0924 11:51:22.727671 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.727682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.728250 (+ 568us) server_negotiation.cc:407] Connection header received
0924 11:51:22.728448 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.728455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.728672 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.728852 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:22.729410 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.730886 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.733841 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.735289 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.735414 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.735604 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.736597 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.736784 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1830,"thread_start_us":1694,"threads_started":1}
I20260924 11:51:22.737973 18020 tablet_service.cc:1467] Processing CreateTablet for tablet 0b857a7eb0354c38a7ad7617b795dde5 (DEFAULT_TABLE table=test-table [id=d8e6014a254a4d63adaefb3db12b3708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:51:22.739450 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b857a7eb0354c38a7ad7617b795dde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:51:22.740204 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 0b857a7eb0354c38a7ad7617b795dde5 (DEFAULT_TABLE table=test-table [id=d8e6014a254a4d63adaefb3db12b3708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:51:22.740657 18152 tablet_service.cc:1467] Processing CreateTablet for tablet 0b857a7eb0354c38a7ad7617b795dde5 (DEFAULT_TABLE table=test-table [id=d8e6014a254a4d63adaefb3db12b3708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:51:22.741400 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b857a7eb0354c38a7ad7617b795dde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:51:22.741864 18152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b857a7eb0354c38a7ad7617b795dde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:51:22.746568 18085 heartbeater.cc:503] Master 127.1.210.62:41355 was elected leader, sending a full tablet report...
I20260924 11:51:22.749692 18364 tablet_bootstrap.cc:492] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Bootstrap starting.
I20260924 11:51:22.753999 18364 tablet_bootstrap.cc:654] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Neither blocks nor log segments found. Creating new log.
I20260924 11:51:22.754710 18365 tablet_bootstrap.cc:492] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Bootstrap starting.
I20260924 11:51:22.755795 18364 log.cc:824] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Log is configured to *not* fsync() on all Append() calls
I20260924 11:51:22.756832 18366 tablet_bootstrap.cc:492] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Bootstrap starting.
I20260924 11:51:22.758013 18365 tablet_bootstrap.cc:654] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Neither blocks nor log segments found. Creating new log.
I20260924 11:51:22.758939 18365 log.cc:824] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Log is configured to *not* fsync() on all Append() calls
I20260924 11:51:22.761585 18366 tablet_bootstrap.cc:654] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Neither blocks nor log segments found. Creating new log.
I20260924 11:51:22.762363 18365 tablet_bootstrap.cc:492] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: No bootstrap required, opened a new log
I20260924 11:51:22.762627 18365 ts_tablet_manager.cc:1397] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260924 11:51:22.763160 18366 log.cc:824] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:51:22.765343 18364 tablet_bootstrap.cc:492] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: No bootstrap required, opened a new log
I20260924 11:51:22.765686 18364 ts_tablet_manager.cc:1397] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260924 11:51:22.766290 18366 tablet_bootstrap.cc:492] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: No bootstrap required, opened a new log
I20260924 11:51:22.766539 18366 ts_tablet_manager.cc:1397] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 11:51:22.770565 18365 raft_consensus.cc:348] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.770910 18365 raft_consensus.cc:374] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:51:22.771015 18365 raft_consensus.cc:729] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6854f47ab74a3d864cff9db11ab268, State: Initialized, Role: FOLLOWER
I20260924 11:51:22.771602 18365 consensus_queue.cc:260] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.772645 18348 heartbeater.cc:503] Master 127.1.210.62:41355 was elected leader, sending a full tablet report...
I20260924 11:51:22.773332 18365 ts_tablet_manager.cc:1428] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260924 11:51:22.776270 18364 raft_consensus.cc:348] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.776723 18364 raft_consensus.cc:374] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:51:22.776824 18364 raft_consensus.cc:729] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997973b2f58f4be2ad3308a17cc6c953, State: Initialized, Role: FOLLOWER
I20260924 11:51:22.776877 18366 raft_consensus.cc:348] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.777345 18366 raft_consensus.cc:374] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:51:22.777446 18366 raft_consensus.cc:729] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909ec80f357c421d88995d11ffba5ca7, State: Initialized, Role: FOLLOWER
I20260924 11:51:22.777483 18364 consensus_queue.cc:260] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.778005 18366 consensus_queue.cc:260] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.781040 18364 ts_tablet_manager.cc:1428] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260924 11:51:22.782076 18217 heartbeater.cc:503] Master 127.1.210.62:41355 was elected leader, sending a full tablet report...
I20260924 11:51:22.782563 18366 ts_tablet_manager.cc:1428] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260924 11:51:22.806722 18371 raft_consensus.cc:482] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:51:22.806984 18371 raft_consensus.cc:504] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.808687 18371 leader_election.cc:288] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525), 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2:38571)
I20260924 11:51:22.815735 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.809162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:50239 (local address 127.1.210.3:44525)
0924 11:51:22.809363 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.809369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.810097 (+ 728us) server_negotiation.cc:407] Connection header received
0924 11:51:22.810414 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.810422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.810513 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.810609 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:22.811132 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.812036 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.813945 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.814776 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.814872 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.814944 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.815336 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.815468 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:51:22.815742 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.809168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:44525 (local address 127.1.210.1:50239)
0924 11:51:22.809972 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.809994 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.810233 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.810669 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.810674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.810713 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:22.810974 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.810982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.812180 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.812185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.813793 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.813805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.814984 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.814991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.815163 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.815198 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.815289 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.815385 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":393,"threads_started":1}
I20260924 11:51:22.815934 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.809144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:54559 (local address 127.1.210.2:38571)
0924 11:51:22.809352 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.809361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.810467 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 11:51:22.810707 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.810713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.810789 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.810847 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:51:22.811575 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.812362 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.814138 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.814984 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.815116 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.815216 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.815552 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.815671 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 11:51:22.816226 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.809168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:38571 (local address 127.1.210.1:54559)
0924 11:51:22.810372 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:22.810389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:51:22.810560 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:51:22.810994 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:51:22.810997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:51:22.811015 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:51:22.811416 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.811430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.812499 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.812503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.813987 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:51:22.814001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.815194 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.815198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:51:22.815383 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.815416 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:51:22.815512 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:51:22.815630 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":207,"thread_start_us":626,"threads_started":1}
I20260924 11:51:22.817117 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b857a7eb0354c38a7ad7617b795dde5" candidate_uuid: "997973b2f58f4be2ad3308a17cc6c953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909ec80f357c421d88995d11ffba5ca7" is_pre_election: true
I20260924 11:51:22.817118 18303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b857a7eb0354c38a7ad7617b795dde5" candidate_uuid: "997973b2f58f4be2ad3308a17cc6c953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6854f47ab74a3d864cff9db11ab268" is_pre_election: true
I20260924 11:51:22.817674 18172 raft_consensus.cc:2454] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 997973b2f58f4be2ad3308a17cc6c953 in term 0.
I20260924 11:51:22.817674 18303 raft_consensus.cc:2454] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 997973b2f58f4be2ad3308a17cc6c953 in term 0.
I20260924 11:51:22.818408 17972 leader_election.cc:302] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f6854f47ab74a3d864cff9db11ab268, 909ec80f357c421d88995d11ffba5ca7, 997973b2f58f4be2ad3308a17cc6c953; no voters:
I20260924 11:51:22.818813 18371 raft_consensus.cc:2781] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:51:22.818940 18371 raft_consensus.cc:482] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:51:22.819078 18371 raft_consensus.cc:3037] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:51:22.821305 18371 raft_consensus.cc:504] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.822009 18371 leader_election.cc:288] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [CANDIDATE]: Term 1 election: Requested vote from peers 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525), 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2:38571)
I20260924 11:51:22.822445 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b857a7eb0354c38a7ad7617b795dde5" candidate_uuid: "997973b2f58f4be2ad3308a17cc6c953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909ec80f357c421d88995d11ffba5ca7"
I20260924 11:51:22.822434 18303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b857a7eb0354c38a7ad7617b795dde5" candidate_uuid: "997973b2f58f4be2ad3308a17cc6c953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6854f47ab74a3d864cff9db11ab268"
I20260924 11:51:22.822644 18303 raft_consensus.cc:3037] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:51:22.822644 18172 raft_consensus.cc:3037] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:51:22.825942 18172 raft_consensus.cc:2454] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997973b2f58f4be2ad3308a17cc6c953 in term 1.
I20260924 11:51:22.825942 18303 raft_consensus.cc:2454] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997973b2f58f4be2ad3308a17cc6c953 in term 1.
I20260924 11:51:22.826334 17974 leader_election.cc:302] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909ec80f357c421d88995d11ffba5ca7, 997973b2f58f4be2ad3308a17cc6c953; no voters:
I20260924 11:51:22.826606 18371 raft_consensus.cc:2781] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:51:22.827093 18371 raft_consensus.cc:686] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 LEADER]: Becoming Leader. State: Replica: 997973b2f58f4be2ad3308a17cc6c953, State: Running, Role: LEADER
I20260924 11:51:22.827557 18371 consensus_queue.cc:237] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } }
I20260924 11:51:22.832226 17899 catalog_manager.cc:5495] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 reported cstate change: term changed from 0 to 1, leader changed from <none> to 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1). New cstate: current_term: 1 leader_uuid: "997973b2f58f4be2ad3308a17cc6c953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997973b2f58f4be2ad3308a17cc6c953" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 } health_report { overall_health: UNKNOWN } } }
I20260924 11:51:22.876333 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.871083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.1.210.1:46141)
0924 11:51:22.871320 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.871326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.871345 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:51:22.871479 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.871483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.871584 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.871734 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:51:22.872098 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.872830 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.874221 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.874626 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.874763 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.875284 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:51:22.875777 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:51:22.875933 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.875997 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.876075 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:51:22.883114 18349 tablet.cc:2369] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:51:22.884503 18303 raft_consensus.cc:1261] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Refusing update from remote peer 997973b2f58f4be2ad3308a17cc6c953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:51:22.884505 18172 raft_consensus.cc:1261] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Refusing update from remote peer 997973b2f58f4be2ad3308a17cc6c953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:51:22.885748 18375 consensus_queue.cc:1035] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6854f47ab74a3d864cff9db11ab268" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:51:22.886132 18371 consensus_queue.cc:1035] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909ec80f357c421d88995d11ffba5ca7" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 38571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:51:22.917245 17898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:51:22.917626 17898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:51:22.921648 17898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Sending DeleteTablet for 3 replicas of tablet 0b857a7eb0354c38a7ad7617b795dde5
I20260924 11:51:22.922820 18020 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b857a7eb0354c38a7ad7617b795dde5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:22 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:38112
I20260924 11:51:22.923017 18283 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b857a7eb0354c38a7ad7617b795dde5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:22 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:60620
I20260924 11:51:22.923179 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17936
I20260924 11:51:22.924068 18152 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b857a7eb0354c38a7ad7617b795dde5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:51:22 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:46550
I20260924 11:51:22.927322 18388 tablet_replica.cc:330] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: stopping tablet replica
I20260924 11:51:22.927876 18388 raft_consensus.cc:2229] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:51:22.928807 18388 raft_consensus.cc:2258] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:51:22.929693 18390 tablet_replica.cc:330] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: stopping tablet replica
I20260924 11:51:22.930142 18390 raft_consensus.cc:2229] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:51:22.930652 18390 raft_consensus.cc:2258] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:51:22.936993 18388 ts_tablet_manager.cc:1905] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:22.941470 18389 tablet_replica.cc:330] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: stopping tablet replica
I20260924 11:51:22.941903 18389 raft_consensus.cc:2229] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:51:22.942438 18389 raft_consensus.cc:2258] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:51:22.942529 18388 ts_tablet_manager.cc:1918] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:51:22.942692 18388 log.cc:1196] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/0b857a7eb0354c38a7ad7617b795dde5
I20260924 11:51:22.943024 18390 ts_tablet_manager.cc:1905] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:22.943356 18388 ts_tablet_manager.cc:1939] T 0b857a7eb0354c38a7ad7617b795dde5 P 997973b2f58f4be2ad3308a17cc6c953: Deleting consensus metadata
I20260924 11:51:22.944514 18389 ts_tablet_manager.cc:1905] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:51:22.945182 17885 catalog_manager.cc:4841] TS 997973b2f58f4be2ad3308a17cc6c953 (127.1.210.1:46141): tablet 0b857a7eb0354c38a7ad7617b795dde5 (table test-table [id=d8e6014a254a4d63adaefb3db12b3708]) successfully deleted
I20260924 11:51:22.948560 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:22.924271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.1.210.1:46141)
0924 11:51:22.924468 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:22.924475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:22.924503 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:51:22.924587 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:22.924593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:22.924680 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:22.924808 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:22.926789 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.931999 (+ 5210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.934716 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:22.935358 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:22.935491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:22.941158 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:22.941173 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:22.941190 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:22.941245 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:22.945233 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.946189 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.946199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.946208 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.946395 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:22.946823 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:22.946828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:22.946831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:22.947266 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:22.947563 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:22.947905 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 11:51:22.948130 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:51:22.949198 18080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:22.952713 18389 ts_tablet_manager.cc:1918] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:51:22.952875 18389 log.cc:1196] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/0b857a7eb0354c38a7ad7617b795dde5
I20260924 11:51:22.953421 18389 ts_tablet_manager.cc:1939] T 0b857a7eb0354c38a7ad7617b795dde5 P 5f6854f47ab74a3d864cff9db11ab268: Deleting consensus metadata
I20260924 11:51:22.953751 18390 ts_tablet_manager.cc:1918] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:51:22.953912 18390 log.cc:1196] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/0b857a7eb0354c38a7ad7617b795dde5
I20260924 11:51:22.954336 18390 ts_tablet_manager.cc:1939] T 0b857a7eb0354c38a7ad7617b795dde5 P 909ec80f357c421d88995d11ffba5ca7: Deleting consensus metadata
I20260924 11:51:22.955387 17883 catalog_manager.cc:4841] TS 5f6854f47ab74a3d864cff9db11ab268 (127.1.210.3:44525): tablet 0b857a7eb0354c38a7ad7617b795dde5 (table test-table [id=d8e6014a254a4d63adaefb3db12b3708]) successfully deleted
I20260924 11:51:22.955870 17885 catalog_manager.cc:4841] TS 909ec80f357c421d88995d11ffba5ca7 (127.1.210.2:38571): tablet 0b857a7eb0354c38a7ad7617b795dde5 (table test-table [id=d8e6014a254a4d63adaefb3db12b3708]) successfully deleted
I20260924 11:51:23.047942 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17936
I20260924 11:51:23.066102 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 18089
I20260924 11:51:23.079824 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:23.066947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.1.210.2:38571)
0924 11:51:23.067125 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:23.067130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:23.067148 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:51:23.067280 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:23.067283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:23.067351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:23.067430 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:23.068809 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:23.069631 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:23.070899 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:23.071273 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:23.071360 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:23.074065 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:23.074076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:23.074091 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:23.074121 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:23.077268 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:23.078012 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:23.078019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:23.078021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:23.078133 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:23.078454 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:23.078458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:23.078459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:23.078809 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:23.079019 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:23.079283 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:51:23.079445 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:51:23.080569 18212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:23.169919 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 18089
I20260924 11:51:23.186749 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 18220
I20260924 11:51:23.200356 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:23.187568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.1.210.3:44525)
0924 11:51:23.187755 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 11:51:23.187763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:51:23.187787 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:51:23.188010 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:51:23.188014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:51:23.188078 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:51:23.188183 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:51:23.189673 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:23.190407 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:23.191750 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:51:23.192143 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:51:23.192280 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:51:23.194908 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:51:23.194922 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:51:23.194937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:51:23.194968 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:51:23.197604 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:23.198310 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:23.198319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:23.198321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:23.198449 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:51:23.198815 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:51:23.198821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:51:23.198825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:51:23.199175 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:51:23.199478 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 11:51:23.199782 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 11:51:23.200010 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:51:23.201025 18343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:23.289259 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 18220
I20260924 11:51:23.306005 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 17866
I20260924 11:51:23.307209 17929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:51:23.419018 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 17866
I20260924 11:51:43.440857 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41355
--webserver_interface=127.1.210.62
--webserver_port=43837
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:51:43.774618 18395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:43.774959 18395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:43.775112 18395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:43.783073 18395 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:51:43.783181 18395 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:51:43.783245 18395 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:51:43.783324 18395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:51:43.783411 18395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:43.783533 18395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:43.783605 18395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:51:43.783694 18395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:51:43.794270 18395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=43837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:43.795775 18395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:43.797283 18395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:43.810247 18401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:43.810258 18400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:43.810479 18403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:43.810882 18395 server_base.cc:1034] running on GCE node
I20260924 11:51:43.811725 18395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:43.813290 18395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:43.814514 18395 hybrid_clock.cc:648] HybridClock initialized: now 1790250703814458 us; error 31 us; skew 500 ppm
I20260924 11:51:43.814864 18395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:43 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250703, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:43.817977 18395 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:51:43.819491 18395 webserver.cc:460] Webserver started at http://127.1.210.62:43837/ using document root <none> and password file <none>
I20260924 11:51:43.820212 18395 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:43.820344 18395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:43.826042 18395 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:51:43.829274 18410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:43.830695 18395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:51:43.830897 18395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "2667fc4a393c48529c9c3ad031a6fe0e"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "a9336c68f37225810e30eec8f586e28c"
server_key_iv: "25cdb024bf0c70421928bf9f2a28e223"
server_key_version: "encryptionkey@0"
I20260924 11:51:43.831839 18395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:43.843206 18395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:43.847142 18395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:43.847436 18395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:43.866133 18395 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:41355
I20260924 11:51:43.866164 18461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:41355 every 8 connection(s)
I20260924 11:51:43.867617 18395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:51:43.873752 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18395
I20260924 11:51:43.874423 18462 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:51:43.874625 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:46141
--local_ip_for_outbound_sockets=127.1.210.1
--tserver_master_addrs=127.1.210.62:41355
--webserver_port=39641
--webserver_interface=127.1.210.1
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:51:43.877849 18462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap starting.
I20260924 11:51:43.898689 18462 log.cc:824] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:51:43.916437 18462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:51:43.917048 18462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Bootstrap complete.
I20260924 11:51:43.926750 18462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:43.927851 18462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2667fc4a393c48529c9c3ad031a6fe0e, State: Initialized, Role: FOLLOWER
I20260924 11:51:43.928481 18462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:43.928756 18462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:51:43.928903 18462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:51:43.929142 18462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:51:43.933634 18462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:43.934170 18462 leader_election.cc:302] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2667fc4a393c48529c9c3ad031a6fe0e; no voters:
I20260924 11:51:43.934931 18462 leader_election.cc:288] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 11:51:43.935165 18466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:51:43.936380 18466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [term 3 LEADER]: Becoming Leader. State: Replica: 2667fc4a393c48529c9c3ad031a6fe0e, State: Running, Role: LEADER
I20260924 11:51:43.937003 18466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } }
I20260924 11:51:43.937387 18462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:51:43.939960 18467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } } }
I20260924 11:51:43.939911 18468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2667fc4a393c48529c9c3ad031a6fe0e. Latest consensus state: current_term: 3 leader_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2667fc4a393c48529c9c3ad031a6fe0e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41355 } } }
I20260924 11:51:43.940357 18467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: This master's current role is: LEADER
I20260924 11:51:43.940435 18468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e [sys.catalog]: This master's current role is: LEADER
I20260924 11:51:43.952102 18471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:51:43.963642 18471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:51:43.965747 18471 catalog_manager.cc:1259] Loaded cluster ID: 98619ab58ba742f59a5091981907a393
I20260924 11:51:43.965888 18471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:51:43.971524 18471 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:51:43.974043 18471 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Loaded TSK: 0
I20260924 11:51:43.974264 18471 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2667fc4a393c48529c9c3ad031a6fe0e: Loaded TSK: 1
F20260924 11:51:43.975889 18471 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260924 11:51:43.978439 18483 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fa3ea8389ad google::LogMessage::Fail() at ??:0
@ 0x7fa3ea83b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa3ea8384de google::LogMessage::Flush() at ??:0
@ 0x7fa3ea839a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa3ee59c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa3ee58b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa4002671c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa400266783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa4002f22a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa4002f20c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa40033c5e9 std::function<>::operator()() at ??:0
@ 0x7fa40026acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa400268c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa4002f0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa4002f0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa3ed484345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa3ed48791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa3ed487799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa3ed462eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa3f073c6db start_thread at ??:0
@ 0x7fa3e96d261f clone at ??:0
W20260924 11:51:44.286350 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:44.286623 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:44.286818 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:44.294216 18464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:44.294353 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:44.294541 18464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:51:44.304793 18464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:46141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=39641
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:44.306181 18464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:44.307613 18464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:44.321272 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:44.321328 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:44.321363 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:44.322196 18464 server_base.cc:1034] running on GCE node
I20260924 11:51:44.322852 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:44.323755 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:44.325003 18464 hybrid_clock.cc:648] HybridClock initialized: now 1790250704324958 us; error 64 us; skew 500 ppm
I20260924 11:51:44.325354 18464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:44 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250704, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:44.328495 18464 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:51:44.330063 18464 webserver.cc:460] Webserver started at http://127.1.210.1:39641/ using document root <none> and password file <none>
I20260924 11:51:44.330744 18464 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:44.330889 18464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:44.336095 18464 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 11:51:44.338945 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:44.340176 18464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:51:44.340387 18464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "997973b2f58f4be2ad3308a17cc6c953"
format_stamp: "Formatted at 2026-09-24 11:50:58 on dist-test-slave-dd32"
server_key: "dd7172f1cf17d474e194dc012a55d777"
server_key_iv: "4ac01cb3d0f39912ce8f8a2cebbf8f8d"
server_key_version: "encryptionkey@0"
I20260924 11:51:44.341419 18464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:44.358708 18464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:44.362455 18464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:44.362718 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:44.363696 18464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:51:44.364998 18464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:51:44.365157 18464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 11:51:44.365291 18464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:51:44.365382 18464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:44.399302 18464 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:46141
I20260924 11:51:44.399395 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:46141 every 8 connection(s)
I20260924 11:51:44.400835 18464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:51:44.405586 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:44.403698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:49035)
0924 11:51:44.404418 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:44.404658 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":295,"threads_started":1}
W20260924 11:51:44.406265 18632 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:41355 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.62:41355: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
I20260924 11:51:44.407096 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:44.406801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:35689)
0924 11:51:44.406937 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:44.406999 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:51:44.407934 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:44.407612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:51655)
0924 11:51:44.407733 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:44.407807 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 11:51:44.408174 18632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:51:44.409520 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18464
I20260924 11:51:44.410118 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:38571
--local_ip_for_outbound_sockets=127.1.210.2
--tserver_master_addrs=127.1.210.62:41355
--webserver_port=40001
--webserver_interface=127.1.210.2
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:51:44.735001 18635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:44.735263 18635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:44.735450 18635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:44.742794 18635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:44.742898 18635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:44.743067 18635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:51:44.752907 18635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:38571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=40001
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:44.754174 18635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:44.755450 18635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:44.768265 18640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:44.768306 18641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:44.768348 18643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:44.769371 18635 server_base.cc:1034] running on GCE node
I20260924 11:51:44.769959 18635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:44.770824 18635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:44.772061 18635 hybrid_clock.cc:648] HybridClock initialized: now 1790250704771993 us; error 50 us; skew 500 ppm
I20260924 11:51:44.772442 18635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:44 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250704, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:44.781590 18635 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:51:44.783185 18635 webserver.cc:460] Webserver started at http://127.1.210.2:40001/ using document root <none> and password file <none>
I20260924 11:51:44.783928 18635 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:44.784104 18635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:44.789929 18635 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:51:44.792943 18650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:44.794368 18635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:51:44.794562 18635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "909ec80f357c421d88995d11ffba5ca7"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "2887ba95e9949aef324a63414af01420"
server_key_iv: "74bb305650b080aaba0c9df76860300c"
server_key_version: "encryptionkey@0"
I20260924 11:51:44.795584 18635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:44.808713 18635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:44.812685 18635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:44.812976 18635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:44.813948 18635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:51:44.815235 18635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:51:44.815351 18635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:44.815480 18635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:51:44.815572 18635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:44.849859 18635 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:38571
I20260924 11:51:44.849920 18762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:38571 every 8 connection(s)
I20260924 11:51:44.851455 18635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:51:44.852943 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18635
I20260924 11:51:44.853830 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:44525
--local_ip_for_outbound_sockets=127.1.210.3
--tserver_master_addrs=127.1.210.62:41355
--webserver_port=35041
--webserver_interface=127.1.210.3
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:51:44.857239 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:44.855190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:37219)
0924 11:51:44.856033 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:44.856221 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":368,"threads_started":1}
W20260924 11:51:44.858084 18763 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:41355 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.62:41355: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
I20260924 11:51:44.859032 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:44.858666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:43473)
0924 11:51:44.858831 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:44.858921 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:51:44.859910 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:44.859579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:55469)
0924 11:51:44.859717 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:44.859799 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:51:44.860121 18763 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:51:45.193645 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:51:45.194008 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:51:45.194193 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:51:45.201942 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:51:45.202123 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:51:45.202316 18766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:51:45.212545 18766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:44525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=35041
--tserver_master_addrs=127.1.210.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:51:45.213985 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:51:45.215427 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:51:45.229539 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:45.229621 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:51:45.229648 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:51:45.230444 18766 server_base.cc:1034] running on GCE node
I20260924 11:51:45.231125 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:51:45.232050 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:51:45.233228 18766 hybrid_clock.cc:648] HybridClock initialized: now 1790250705233162 us; error 56 us; skew 500 ppm
I20260924 11:51:45.233604 18766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:51:45 dist-test-slave-dd32 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250705, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:51:45.236835 18766 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:51:45.238440 18766 webserver.cc:460] Webserver started at http://127.1.210.3:35041/ using document root <none> and password file <none>
I20260924 11:51:45.239166 18766 fs_manager.cc:359] Metadata directory not provided
I20260924 11:51:45.239318 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:51:45.244823 18766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:51:45.247695 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:45.249032 18766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:51:45.249313 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "5f6854f47ab74a3d864cff9db11ab268"
format_stamp: "Formatted at 2026-09-24 11:50:59 on dist-test-slave-dd32"
server_key: "b59a22382546362134b660bf1ad71adc"
server_key_iv: "c601fb6c8737912d2ccbc3fe06e6ee49"
server_key_version: "encryptionkey@0"
I20260924 11:51:45.250293 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:51:45.273289 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:51:45.277566 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:51:45.277880 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:51:45.278924 18766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:51:45.280493 18766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:51:45.280622 18766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:45.280776 18766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:51:45.280874 18766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:51:45.314339 18766 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:44525
I20260924 11:51:45.314363 18893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:44525 every 8 connection(s)
I20260924 11:51:45.315817 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:51:45.319068 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18766
I20260924 11:51:45.320957 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:45.318853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:54143)
0924 11:51:45.319679 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:45.319972 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":363,"threads_started":1}
W20260924 11:51:45.321544 18894 heartbeater.cc:644] Failed to heartbeat to 127.1.210.62:41355 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.62:41355: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
I20260924 11:51:45.322343 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:45.322007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:35581)
0924 11:51:45.322138 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:45.322201 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 11:51:45.323073 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:45.322783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:59493)
0924 11:51:45.322901 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:45.322968 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:51:45.323323 18894 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:51:45.409624 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:45.408739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:46999)
0924 11:51:45.409299 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:45.409404 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":295,"threads_started":1}
I20260924 11:51:45.861510 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:45.860670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:49955)
0924 11:51:45.861166 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:45.861275 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260924 11:51:46.324625 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:46.323845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:49571)
0924 11:51:46.324303 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:46.324389 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260924 11:51:46.411178 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:46.410493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:37021)
0924 11:51:46.410932 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:46.411014 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260924 11:51:46.863040 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:46.862357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:50725)
0924 11:51:46.862796 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:46.862875 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260924 11:51:47.326637 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:47.325604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:57527)
0924 11:51:47.326322 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:47.326472 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":465,"threads_started":1}
I20260924 11:51:47.412729 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:47.412034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:47357)
0924 11:51:47.412480 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:47.412559 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260924 11:51:47.864599 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:47.863902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:53485)
0924 11:51:47.864353 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:47.864437 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260924 11:51:48.328157 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:48.327465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:50257)
0924 11:51:48.327899 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:48.327981 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260924 11:51:48.414258 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:48.413565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:54679)
0924 11:51:48.413987 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:48.414067 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":206,"threads_started":1}
I20260924 11:51:48.866500 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:48.865522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:53159)
0924 11:51:48.866148 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:48.866257 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":301,"threads_started":1}
I20260924 11:51:49.329818 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:49.329140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:60797)
0924 11:51:49.329550 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:49.329628 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":200,"threads_started":1}
I20260924 11:51:49.415907 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:49.415185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:37721)
0924 11:51:49.415657 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:49.415742 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260924 11:51:49.868194 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:49.867378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:55025)
0924 11:51:49.867888 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:49.867980 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
I20260924 11:51:50.331410 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:50.330706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:48069)
0924 11:51:50.331162 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:50.331244 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260924 11:51:50.417748 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:50.416898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:38303)
0924 11:51:50.417490 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:50.417582 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":359,"threads_started":1}
I20260924 11:51:50.869884 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:50.869146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:38175)
0924 11:51:50.869628 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:50.869727 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":241,"threads_started":1}
I20260924 11:51:51.332947 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:51.332260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:34003)
0924 11:51:51.332696 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:51.332787 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260924 11:51:51.419497 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:51.418676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:34891)
0924 11:51:51.419177 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:51.419257 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":233,"threads_started":1}
I20260924 11:51:51.871536 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:51.870737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:41443)
0924 11:51:51.871240 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:51.871321 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":227,"threads_started":1}
I20260924 11:51:52.334556 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:52.333793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:57953)
0924 11:51:52.334251 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:52.334329 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":208,"threads_started":1}
I20260924 11:51:52.421576 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:52.420576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:35877)
0924 11:51:52.421242 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:52.421368 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":385,"threads_started":1}
I20260924 11:51:52.873518 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:52.872735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:43037)
0924 11:51:52.873270 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:52.873356 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":253,"threads_started":1}
I20260924 11:51:53.336087 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:53.335396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:44729)
0924 11:51:53.335847 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:53.335928 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260924 11:51:53.423107 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:53.422435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:47667)
0924 11:51:53.422868 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:53.422944 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":212,"threads_started":1}
I20260924 11:51:53.875243 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:53.874506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:55939)
0924 11:51:53.874951 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:53.875043 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":225,"threads_started":1}
I20260924 11:51:54.337746 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:54.336934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:38377)
0924 11:51:54.337505 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:54.337579 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":334,"threads_started":1}
I20260924 11:51:54.424785 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:54.424051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:39283)
0924 11:51:54.424535 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:54.424610 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":278,"threads_started":1}
I20260924 11:51:54.876995 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:54.876316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:55561)
0924 11:51:54.876746 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:54.876824 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":241,"threads_started":1}
I20260924 11:51:55.339359 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:55.338625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:37633)
0924 11:51:55.339108 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:55.339195 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260924 11:51:55.426414 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:55.425685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:54969)
0924 11:51:55.426147 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:55.426238 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260924 11:51:55.878633 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:55.877907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:36405)
0924 11:51:55.878371 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:55.878459 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260924 11:51:56.340922 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:56.340245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:47289)
0924 11:51:56.340656 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:56.340748 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":203,"threads_started":1}
I20260924 11:51:56.428017 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:56.427343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:48257)
0924 11:51:56.427771 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:56.427849 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":227,"threads_started":1}
I20260924 11:51:56.880715 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:56.879848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:42195)
0924 11:51:56.880400 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:56.880482 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":322,"threads_started":1}
I20260924 11:51:57.342525 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:57.341857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:47083)
0924 11:51:57.342279 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:57.342356 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":224,"threads_started":1}
I20260924 11:51:57.429838 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:57.429018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:53357)
0924 11:51:57.429555 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:57.429649 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":290,"threads_started":1}
I20260924 11:51:57.882308 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:57.881599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:47123)
0924 11:51:57.882012 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:57.882140 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260924 11:51:58.344321 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:58.343366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:49505)
0924 11:51:58.344029 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:58.344140 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":359,"threads_started":1}
I20260924 11:51:58.431419 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:58.430737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:55685)
0924 11:51:58.431164 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:58.431241 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260924 11:51:58.884029 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:58.883284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:57039)
0924 11:51:58.883754 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:58.883846 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
I20260924 11:51:59.345971 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:59.345179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:46977)
0924 11:51:59.345588 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:59.345717 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":199,"threads_started":1}
I20260924 11:51:59.433462 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:59.432590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.1:54951)
0924 11:51:59.433176 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:59.433274 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260924 11:51:59.885996 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:51:59.885294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.2:57327)
0924 11:51:59.885726 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 11:51:59.885826 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":241,"threads_started":1}
I20260924 11:52:00.323971 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 18464
I20260924 11:52:00.341621 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:00.325130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.1.210.1:46141)
0924 11:52:00.326051 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:00.326071 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:00.326123 (+ 52us) server_negotiation.cc:407] Connection header received
0924 11:52:00.327763 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:00.327771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:00.328277 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:00.328576 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:00.329989 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:00.331223 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:00.332516 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:00.333233 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:00.333365 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:00.335765 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:00.335773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:00.335788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:00.335823 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:00.338817 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:00.339411 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:00.339417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:00.339423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:00.339581 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:00.339891 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:00.339894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:00.339896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:00.340379 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:00.340601 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:00.340938 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 11:52:00.341345 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":524,"threads_started":1}
I20260924 11:52:00.342972 18627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:00.347895 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:00.347100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:41355 (local address 127.1.210.3:56901)
0924 11:52:00.347561 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:00.347707 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.62:41355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":252,"threads_started":1}
I20260924 11:52:00.428663 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 18464
I20260924 11:52:00.446036 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 18635
I20260924 11:52:00.463817 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:00.447007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.1.210.2:38571)
0924 11:52:00.447560 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:00.447573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:00.447608 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:52:00.449182 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:00.449193 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:00.449616 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:00.449970 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:00.451432 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:00.452681 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:00.454351 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:00.455033 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:00.455245 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:00.457806 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:00.457814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:00.457829 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:00.457864 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:00.460449 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:00.461281 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:00.461289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:00.461303 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:00.461480 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:00.461885 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:00.461889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:00.461891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:00.462387 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:00.462717 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:00.463132 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 11:52:00.463541 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":239,"threads_started":1}
I20260924 11:52:00.465232 18758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:00.548322 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 18635
I20260924 11:52:00.565680 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 18766
I20260924 11:52:00.582973 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:00.566772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.1.210.3:44525)
0924 11:52:00.567384 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:00.567399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:00.567441 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:52:00.569021 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:00.569054 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:00.569566 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:00.569861 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:00.571093 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:00.572309 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:00.573586 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:00.574245 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:00.574379 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:00.576847 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:00.576857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:00.576876 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:00.576917 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:00.580097 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:00.580668 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:00.580673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:00.580679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:00.580832 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:00.581206 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:00.581209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:00.581211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:00.581687 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:00.581907 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:00.582252 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 11:52:00.582671 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":303,"threads_started":1}
I20260924 11:52:00.584349 18889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:00.667536 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 18766
2026-09-24T11:52:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64367 ms)
[----------] 30 tests from SecurityITest (291290 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:00 dist-test-slave-dd32 krb5kdc[18951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:00 dist-test-slave-dd32 krb5kdc[18951](info): set up 2 sockets
Sep 24 11:52:00 dist-test-slave-dd32 krb5kdc[18951](info): commencing operation
krb5kdc: starting...
W20260924 11:52:01.797155 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:01 dist-test-slave-dd32 krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:02.045035 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:44409
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:02.382230 18964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:02.382527 18964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:02.382616 18964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:02.389910 18964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:02.390019 18964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:02.390066 18964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:02.390116 18964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:02.390161 18964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:02.400234 18964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:44409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:02.401659 18964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:02.402901 18964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:02.415050 18972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:02.415154 18970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:02.415198 18969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:02.416172 18964 server_base.cc:1034] running on GCE node
I20260924 11:52:02.417129 18964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:02.418704 18964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:02.419927 18964 hybrid_clock.cc:648] HybridClock initialized: now 1790250722419874 us; error 43 us; skew 500 ppm
I20260924 11:52:02.420228 18964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:02 dist-test-slave-dd32 krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:02.423367 18964 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:02.424954 18964 webserver.cc:460] Webserver started at http://127.1.210.62:42721/ using document root <none> and password file <none>
I20260924 11:52:02.425693 18964 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:02.425797 18964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:02.426046 18964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:02.428992 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "b57b267a25d545a795fb17f48b42162e"
format_stamp: "Formatted at 2026-09-24 11:52:02 on dist-test-slave-dd32"
server_key: "207fd66907b546e8fff071c8a4732bc2"
server_key_iv: "99d8a0d3d511b9afc0f56b46998c397e"
server_key_version: "encryptionkey@0"
I20260924 11:52:02.429762 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "b57b267a25d545a795fb17f48b42162e"
format_stamp: "Formatted at 2026-09-24 11:52:02 on dist-test-slave-dd32"
server_key: "207fd66907b546e8fff071c8a4732bc2"
server_key_iv: "99d8a0d3d511b9afc0f56b46998c397e"
server_key_version: "encryptionkey@0"
I20260924 11:52:02.435012 18964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:52:02.438599 18979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:02.439966 18964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:02.440143 18964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "b57b267a25d545a795fb17f48b42162e"
format_stamp: "Formatted at 2026-09-24 11:52:02 on dist-test-slave-dd32"
server_key: "207fd66907b546e8fff071c8a4732bc2"
server_key_iv: "99d8a0d3d511b9afc0f56b46998c397e"
server_key_version: "encryptionkey@0"
I20260924 11:52:02.440337 18964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:02.482784 18964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:02.486851 18964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:02.487154 18964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:02.505873 18964 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:44409
I20260924 11:52:02.505890 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:44409 every 8 connection(s)
I20260924 11:52:02.507340 18964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:02.509516 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 18964
I20260924 11:52:02.509724 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:02.510354 1864 external_mini_cluster.cc:1428] Setting key 0a55fc432d9f6cc2d5da5be28e5901e8
I20260924 11:52:02.511255 19031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:52:02 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:02.525256 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e: Bootstrap starting.
I20260924 11:52:02.529348 19031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:02.530616 19031 log.cc:824] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:02.534179 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e: No bootstrap required, opened a new log
I20260924 11:52:02.534693 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:02.512268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.1.210.62:44409)
0924 11:52:02.513016 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:02.513027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:02.513112 (+ 85us) server_negotiation.cc:407] Connection header received
0924 11:52:02.514652 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:02.514671 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:02.515054 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:02.515370 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:02.517251 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:02.519309 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:02.520707 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:02.521816 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:02.522003 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:02.526087 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:02.526097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:02.526119 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:02.526163 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:02.530090 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:02.530840 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:02.530849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:02.530856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:02.531066 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:02.531518 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:02.531521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:02.531523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:02.532161 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:02.532457 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:02.532856 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 11:52:02.533342 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":346,"threads_started":1}
I20260924 11:52:02.542723 19031 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b267a25d545a795fb17f48b42162e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44409 } }
I20260924 11:52:02.543169 19031 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:02.543253 19031 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57b267a25d545a795fb17f48b42162e, State: Initialized, Role: FOLLOWER
I20260924 11:52:02.543782 19031 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b267a25d545a795fb17f48b42162e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44409 } }
I20260924 11:52:02.543978 19031 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:02.544150 19031 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:02.544320 19031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:02.546634 19031 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b267a25d545a795fb17f48b42162e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44409 } }
I20260924 11:52:02.547247 19031 leader_election.cc:302] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57b267a25d545a795fb17f48b42162e; no voters:
I20260924 11:52:02.547817 19031 leader_election.cc:288] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:02.547966 19036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:02.548779 19036 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [term 1 LEADER]: Becoming Leader. State: Replica: b57b267a25d545a795fb17f48b42162e, State: Running, Role: LEADER
I20260924 11:52:02.549367 19036 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b267a25d545a795fb17f48b42162e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44409 } }
I20260924 11:52:02.550055 19031 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:02.552269 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57b267a25d545a795fb17f48b42162e. Latest consensus state: current_term: 1 leader_uuid: "b57b267a25d545a795fb17f48b42162e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b267a25d545a795fb17f48b42162e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44409 } } }
I20260924 11:52:02.553544 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:02.553431 19037 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57b267a25d545a795fb17f48b42162e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b267a25d545a795fb17f48b42162e" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 44409 } } }
I20260924 11:52:02.554790 19037 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:02.557911 19045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:02.562898 19045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:02.572497 19045 catalog_manager.cc:1347] Generated new cluster ID: 4ed5147d57be4707b51fda8c8e8ac8e9
I20260924 11:52:02.572606 19045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:02.596223 19045 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:02.598234 19045 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:02.605463 19045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e: Generated new TSK 0
I20260924 11:52:02.606460 19045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:52:02.672465 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:44409
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:03.008562 19057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:03.008882 19057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:03.009102 19057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:03.016422 19057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:03.016582 19057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:03.016772 19057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:52:03.026810 19057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:03.028101 19057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:03.029466 19057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:03.042990 19062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:03.042948 19063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:03.042912 19065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:03.043960 19057 server_base.cc:1034] running on GCE node
I20260924 11:52:03.044665 19057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:03.045675 19057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:03.046873 19057 hybrid_clock.cc:648] HybridClock initialized: now 1790250723046835 us; error 29 us; skew 500 ppm
I20260924 11:52:03.047214 19057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:03 dist-test-slave-dd32 krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250723, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:03.050345 19057 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:52:03.051865 19057 webserver.cc:460] Webserver started at http://127.1.210.1:45345/ using document root <none> and password file <none>
I20260924 11:52:03.052551 19057 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:03.052702 19057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:03.053020 19057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:03.056205 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "c21fa3cb898d4b6a95318a0a5f9698c3"
format_stamp: "Formatted at 2026-09-24 11:52:03 on dist-test-slave-dd32"
server_key: "0c002acc628f43576bb7de5b8aa6e514"
server_key_iv: "1c536346500ce4619a10f99460fb9130"
server_key_version: "encryptionkey@0"
I20260924 11:52:03.056959 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "c21fa3cb898d4b6a95318a0a5f9698c3"
format_stamp: "Formatted at 2026-09-24 11:52:03 on dist-test-slave-dd32"
server_key: "0c002acc628f43576bb7de5b8aa6e514"
server_key_iv: "1c536346500ce4619a10f99460fb9130"
server_key_version: "encryptionkey@0"
I20260924 11:52:03.061940 19057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:52:03.065232 19072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:03.066450 19057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:52:03.066634 19057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "c21fa3cb898d4b6a95318a0a5f9698c3"
format_stamp: "Formatted at 2026-09-24 11:52:03 on dist-test-slave-dd32"
server_key: "0c002acc628f43576bb7de5b8aa6e514"
server_key_iv: "1c536346500ce4619a10f99460fb9130"
server_key_version: "encryptionkey@0"
I20260924 11:52:03.066833 19057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:03.103029 19057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:03.107791 19057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:03.108114 19057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:03.109201 19057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:03.110669 19057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:03.110798 19057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 11:52:03.110954 19057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:03.111029 19057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:03.160679 19057 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:40503
I20260924 11:52:03.160759 19184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:40503 every 8 connection(s)
I20260924 11:52:03.162382 19057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:52:03.170006 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19057
I20260924 11:52:03.170202 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:52:03.171041 1864 external_mini_cluster.cc:1428] Setting key 262a00e648a5697d419df471a08ccf3e
Sep 24 11:52:03 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250723, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:03.186683 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:03.165197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:59355 (local address 127.1.210.62:44409)
0924 11:52:03.165676 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:03.165681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:03.166715 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 11:52:03.168547 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:03.168562 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:03.168650 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:03.168774 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:03.170852 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.172248 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.174737 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.175163 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.175267 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:03.180487 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:03.180492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:03.180495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:03.180527 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:03.183124 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:03.184179 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:03.184187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:03.184192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:03.184354 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:03.184902 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:03.184905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:03.184907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:03.185258 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:03.185373 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:03.186264 (+ 891us) server_negotiation.cc:299] Negotiation successful
0924 11:52:03.186430 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
I20260924 11:52:03.188284 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:03.165822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:44409 (local address 127.1.210.1:59355)
0924 11:52:03.166548 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:03.166595 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:03.168282 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:03.169005 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:03.169013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:03.169744 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:03.170646 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:03.170673 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.172431 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.172439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:03.174567 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:03.174598 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.175401 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.175411 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:03.175749 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:03.176655 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:03.176688 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:03.180206 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:03.183336 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:03.183347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:03.183367 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:03.184004 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:03.184516 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:03.184524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:03.184531 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:03.184778 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:03.185399 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:03.185410 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:03.185862 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 11:52:03.186166 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 11:52:03.186792 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":309,"threads_started":1}
I20260924 11:52:03.189910 19185 heartbeater.cc:347] Connected to a master server at 127.1.210.62:44409
I20260924 11:52:03.190227 19185 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:03.190927 19185 heartbeater.cc:511] Master 127.1.210.62:44409 requested a full tablet report, sending...
I20260924 11:52:03.192978 18996 ts_manager.cc:194] Registered new tserver with Master: c21fa3cb898d4b6a95318a0a5f9698c3 (127.1.210.1:40503)
I20260924 11:52:03.194489 18996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:59355
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:52:03.232182 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:44409
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:03.558094 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:03.558353 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:03.558552 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:03.565685 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:03.565814 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:03.566006 19191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:52:03.575920 19191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:03.577332 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:03.578706 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:03.591893 19196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:03.591917 19199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:03.591846 19197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:03.592752 19191 server_base.cc:1034] running on GCE node
I20260924 11:52:03.593372 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:03.594374 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:03.595570 19191 hybrid_clock.cc:648] HybridClock initialized: now 1790250723595509 us; error 55 us; skew 500 ppm
I20260924 11:52:03.595884 19191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:03 dist-test-slave-dd32 krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250723, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:03.599043 19191 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:52:03.600361 19191 webserver.cc:460] Webserver started at http://127.1.210.2:41551/ using document root <none> and password file <none>
I20260924 11:52:03.600976 19191 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:03.601131 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:03.601411 19191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:03.604154 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "e668e655fd694dc19fbd668916f2cd23"
format_stamp: "Formatted at 2026-09-24 11:52:03 on dist-test-slave-dd32"
server_key: "58a24f2eee2ec68b9493d3d9e7f8a5de"
server_key_iv: "260904952e655d030c2033736b9a9932"
server_key_version: "encryptionkey@0"
I20260924 11:52:03.604871 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "e668e655fd694dc19fbd668916f2cd23"
format_stamp: "Formatted at 2026-09-24 11:52:03 on dist-test-slave-dd32"
server_key: "58a24f2eee2ec68b9493d3d9e7f8a5de"
server_key_iv: "260904952e655d030c2033736b9a9932"
server_key_version: "encryptionkey@0"
I20260924 11:52:03.609820 19191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 11:52:03.612960 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:03.614259 19191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:52:03.614434 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "e668e655fd694dc19fbd668916f2cd23"
format_stamp: "Formatted at 2026-09-24 11:52:03 on dist-test-slave-dd32"
server_key: "58a24f2eee2ec68b9493d3d9e7f8a5de"
server_key_iv: "260904952e655d030c2033736b9a9932"
server_key_version: "encryptionkey@0"
I20260924 11:52:03.614634 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:03.634943 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:03.638963 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:03.639238 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:03.640203 19191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:03.641511 19191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:03.641620 19191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:03.641762 19191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:03.641835 19191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:03.675266 19191 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:33525
I20260924 11:52:03.675314 19318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:33525 every 8 connection(s)
I20260924 11:52:03.676618 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:52:03.680080 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19191
I20260924 11:52:03.680275 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:52:03.680740 1864 external_mini_cluster.cc:1428] Setting key 72886504c404eca1beb9f9f3cdd28ff4
Sep 24 11:52:03 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250723, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:03.696810 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:03.679238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:59179 (local address 127.1.210.62:44409)
0924 11:52:03.679423 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:03.679428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:03.680747 (+ 1319us) server_negotiation.cc:407] Connection header received
0924 11:52:03.683003 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:03.683008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:03.683099 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:03.683212 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:03.684679 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.685454 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.686967 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.687406 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.687540 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:03.691092 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:03.691097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:03.691100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:03.691131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:03.693755 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:03.694482 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:03.694488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:03.694491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:03.694629 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:03.695070 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:03.695075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:03.695078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:03.695352 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:03.695514 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:03.696224 (+ 710us) server_negotiation.cc:299] Negotiation successful
0924 11:52:03.696454 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:52:03.697890 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:03.679739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:44409 (local address 127.1.210.2:59179)
0924 11:52:03.680588 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:03.680633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:03.682731 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:03.683446 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:03.683453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:03.683947 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:03.684510 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:03.684527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.685604 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.685610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:03.686823 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:03.686834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:03.687639 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:03.687646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:03.687872 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:03.688556 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:03.688580 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:03.690869 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:03.693899 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:03.693908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:03.693925 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:03.694355 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:03.694744 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:03.694750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:03.694753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:03.694945 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:03.695485 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:03.695493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:03.695851 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 11:52:03.696136 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 11:52:03.696699 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":384,"threads_started":1}
I20260924 11:52:03.699396 19319 heartbeater.cc:347] Connected to a master server at 127.1.210.62:44409
I20260924 11:52:03.699752 19319 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:03.700743 19319 heartbeater.cc:511] Master 127.1.210.62:44409 requested a full tablet report, sending...
I20260924 11:52:03.702556 18996 ts_manager.cc:194] Registered new tserver with Master: e668e655fd694dc19fbd668916f2cd23 (127.1.210.2:33525)
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:52:03.703850 18996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:59179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:52:03.735621 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:44409
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:04.056432 19324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:04.056737 19324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:04.056903 19324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:04.064069 19324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:04.064244 19324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:04.064436 19324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:52:04.074203 19324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:04.075501 19324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:04.076822 19324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:04.089530 19332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:04.089568 19329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:04.089637 19330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:04.090382 19324 server_base.cc:1034] running on GCE node
I20260924 11:52:04.091030 19324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:04.091912 19324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:04.093144 19324 hybrid_clock.cc:648] HybridClock initialized: now 1790250724093096 us; error 43 us; skew 500 ppm
I20260924 11:52:04.093441 19324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:04 dist-test-slave-dd32 krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250724, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:04.096378 19324 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:52:04.097849 19324 webserver.cc:460] Webserver started at http://127.1.210.3:46643/ using document root <none> and password file <none>
I20260924 11:52:04.098513 19324 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:04.098657 19324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:04.098984 19324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:04.101792 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "e872c63b2d6745089da757483666516d"
format_stamp: "Formatted at 2026-09-24 11:52:04 on dist-test-slave-dd32"
server_key: "cc671a9c5262c1e55171c4079f6bb1f3"
server_key_iv: "277e4cb33fdd6d0b3edfffe295a40174"
server_key_version: "encryptionkey@0"
I20260924 11:52:04.102583 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "e872c63b2d6745089da757483666516d"
format_stamp: "Formatted at 2026-09-24 11:52:04 on dist-test-slave-dd32"
server_key: "cc671a9c5262c1e55171c4079f6bb1f3"
server_key_iv: "277e4cb33fdd6d0b3edfffe295a40174"
server_key_version: "encryptionkey@0"
I20260924 11:52:04.107493 19324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:52:04.111243 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:04.113247 19324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 11:52:04.113377 19324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "e872c63b2d6745089da757483666516d"
format_stamp: "Formatted at 2026-09-24 11:52:04 on dist-test-slave-dd32"
server_key: "cc671a9c5262c1e55171c4079f6bb1f3"
server_key_iv: "277e4cb33fdd6d0b3edfffe295a40174"
server_key_version: "encryptionkey@0"
I20260924 11:52:04.113535 19324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:04.125785 19324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:04.129479 19324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:04.129705 19324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:04.130551 19324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:04.131803 19324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:04.131873 19324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:04.131953 19324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:04.132001 19324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:04.164409 19324 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:35037
I20260924 11:52:04.164444 19451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:35037 every 8 connection(s)
I20260924 11:52:04.165709 19324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:52:04.167850 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19324
I20260924 11:52:04.168017 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:52:04.168491 1864 external_mini_cluster.cc:1428] Setting key e64d30b67848ebcf7b5bee2db5419bd9
Sep 24 11:52:04 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250724, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:04.185534 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.169361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:60811 (local address 127.1.210.62:44409)
0924 11:52:04.170101 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.170107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.170126 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:52:04.172016 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.172020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.172110 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.172197 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:04.173687 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.174419 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.175866 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.176250 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.176340 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.179602 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:04.179607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:04.179610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:04.179642 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:04.182435 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.183331 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.183338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.183343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.183487 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.184002 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.184008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.184012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.184317 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:04.184430 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.185113 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.185286 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635}
I20260924 11:52:04.186446 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.168767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:44409 (local address 127.1.210.3:60811)
0924 11:52:04.169601 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:04.169671 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:04.171746 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:04.172391 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:04.172399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:04.172908 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:04.173536 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.173553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.174550 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.174553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.175745 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.175756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.176427 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.176431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.176615 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.177295 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:04.177316 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:04.179415 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:04.182569 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:04.182576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:04.182592 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:04.183188 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:04.183634 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:04.183637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:04.183640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:04.183865 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:04.184431 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:04.184437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:04.184744 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 11:52:04.184955 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 11:52:04.185394 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":332,"threads_started":1}
I20260924 11:52:04.187840 19452 heartbeater.cc:347] Connected to a master server at 127.1.210.62:44409
I20260924 11:52:04.188102 19452 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:04.188758 19452 heartbeater.cc:511] Master 127.1.210.62:44409 requested a full tablet report, sending...
I20260924 11:52:04.190135 18996 ts_manager.cc:194] Registered new tserver with Master: e872c63b2d6745089da757483666516d (127.1.210.3:35037)
I20260924 11:52:04.190950 18996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:60811
I20260924 11:52:04.196007 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:52:04.198290 19185 heartbeater.cc:503] Master 127.1.210.62:44409 was elected leader, sending a full tablet report...
I20260924 11:52:04.212518 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.200811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.1.210.62:44409)
0924 11:52:04.200980 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.200985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.201413 (+ 428us) server_negotiation.cc:407] Connection header received
0924 11:52:04.201541 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.201545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.201603 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.201684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:04.202965 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.203665 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.205030 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.205497 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.205594 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.207744 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:04.207749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:04.207753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:04.207781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:04.210372 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.210936 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.210939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.210941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.211067 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.211361 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.211364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.211366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.211651 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:04.211857 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.212129 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.212279 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:52:04.216355 18996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:52:04.223935 18996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:52:04.248629 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.236310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.1.210.1:40503)
0924 11:52:04.237767 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.237779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.239797 (+ 2018us) server_negotiation.cc:407] Connection header received
0924 11:52:04.240077 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.240084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.240252 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.240422 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:04.242038 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.243476 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.246055 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.247464 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.247611 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.247847 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.248155 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.248305 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":1020,"threads_started":1}
I20260924 11:52:04.248749 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.235786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40503 (local address 127.0.0.1:44650)
0924 11:52:04.239694 (+ 3908us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:04.239710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:04.239898 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:04.240736 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:04.240744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:04.240799 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:04.241815 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.241830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.243622 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.243626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.245894 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.245906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.247683 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.247689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.247891 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.247950 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 11:52:04.248083 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 11:52:04.248265 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3795,"mutex_wait_us":522,"spinlock_wait_cycles":4736,"thread_start_us":2893,"threads_started":1}
I20260924 11:52:04.250926 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.235658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:35037 (local address 127.0.0.1:47516)
0924 11:52:04.239390 (+ 3732us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:04.239434 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:04.239652 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:04.240736 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:04.240744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:04.240799 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:04.241328 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.241342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.243268 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.243273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.246248 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.246263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.248015 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.248025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.250265 (+ 2240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.250296 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:52:04.250380 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 11:52:04.250474 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3146,"mutex_wait_us":100,"spinlock_wait_cycles":7296,"thread_start_us":3742,"threads_started":1}
I20260924 11:52:04.251055 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.239369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.1.210.2:33525)
0924 11:52:04.241453 (+ 2084us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.241463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.242760 (+ 1297us) server_negotiation.cc:407] Connection header received
0924 11:52:04.242905 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.242913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.243165 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.243347 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:04.245189 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.246627 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.248592 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.250000 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.250147 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.250326 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.250583 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.250747 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":1803,"threads_started":1}
I20260924 11:52:04.251351 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.238747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:33525 (local address 127.0.0.1:56046)
0924 11:52:04.242621 (+ 3874us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:04.242639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:04.242754 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:04.243356 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:04.243362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:04.243389 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:04.244904 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.244920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.246759 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.246766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.248429 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.248443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.250219 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.250225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.250390 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.250422 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:52:04.250516 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 11:52:04.250689 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3732,"mutex_wait_us":52,"thread_start_us":3359,"threads_started":1}
I20260924 11:52:04.251727 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.235938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.1.210.3:35037)
0924 11:52:04.238361 (+ 2423us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.238372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.239846 (+ 1474us) server_negotiation.cc:407] Connection header received
0924 11:52:04.239960 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.239969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.240248 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.240429 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:04.241546 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.243083 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.246423 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.247804 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.247937 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.248182 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.251196 (+ 3014us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.251387 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2215,"thread_start_us":2022,"threads_started":1}
I20260924 11:52:04.252946 19120 tablet_service.cc:1467] Processing CreateTablet for tablet 86004f16e61a4c498b6eba230af9656f (DEFAULT_TABLE table=test-table [id=a48899e7aadc42608e5f0de91b995add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:04.255517 19120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86004f16e61a4c498b6eba230af9656f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:04.255323 19254 tablet_service.cc:1467] Processing CreateTablet for tablet 86004f16e61a4c498b6eba230af9656f (DEFAULT_TABLE table=test-table [id=a48899e7aadc42608e5f0de91b995add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:04.255595 19387 tablet_service.cc:1467] Processing CreateTablet for tablet 86004f16e61a4c498b6eba230af9656f (DEFAULT_TABLE table=test-table [id=a48899e7aadc42608e5f0de91b995add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:04.256624 19254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86004f16e61a4c498b6eba230af9656f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:04.256842 19387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86004f16e61a4c498b6eba230af9656f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:04.268042 19470 tablet_bootstrap.cc:492] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Bootstrap starting.
I20260924 11:52:04.271265 19472 tablet_bootstrap.cc:492] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Bootstrap starting.
I20260924 11:52:04.271396 19470 tablet_bootstrap.cc:654] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:04.271608 19471 tablet_bootstrap.cc:492] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Bootstrap starting.
I20260924 11:52:04.272692 19470 log.cc:824] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:04.274891 19471 tablet_bootstrap.cc:654] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:04.276093 19470 tablet_bootstrap.cc:492] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: No bootstrap required, opened a new log
I20260924 11:52:04.276093 19471 log.cc:824] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:04.276340 19472 tablet_bootstrap.cc:654] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:04.276445 19470 ts_tablet_manager.cc:1397] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 11:52:04.277686 19472 log.cc:824] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:04.279699 19471 tablet_bootstrap.cc:492] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: No bootstrap required, opened a new log
I20260924 11:52:04.279949 19471 ts_tablet_manager.cc:1397] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 11:52:04.280364 19472 tablet_bootstrap.cc:492] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: No bootstrap required, opened a new log
I20260924 11:52:04.280628 19472 ts_tablet_manager.cc:1397] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 11:52:04.287278 19470 raft_consensus.cc:348] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.287695 19470 raft_consensus.cc:374] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:04.287830 19470 raft_consensus.cc:729] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e872c63b2d6745089da757483666516d, State: Initialized, Role: FOLLOWER
I20260924 11:52:04.288446 19470 consensus_queue.cc:260] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.288941 19472 raft_consensus.cc:348] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.289480 19472 raft_consensus.cc:374] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:04.289628 19452 heartbeater.cc:503] Master 127.1.210.62:44409 was elected leader, sending a full tablet report...
I20260924 11:52:04.289638 19472 raft_consensus.cc:729] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e668e655fd694dc19fbd668916f2cd23, State: Initialized, Role: FOLLOWER
I20260924 11:52:04.290231 19470 ts_tablet_manager.cc:1428] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 11:52:04.290251 19471 raft_consensus.cc:348] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.290439 19472 consensus_queue.cc:260] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.290676 19471 raft_consensus.cc:374] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:04.290838 19471 raft_consensus.cc:729] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21fa3cb898d4b6a95318a0a5f9698c3, State: Initialized, Role: FOLLOWER
I20260924 11:52:04.291394 19471 consensus_queue.cc:260] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.292022 19319 heartbeater.cc:503] Master 127.1.210.62:44409 was elected leader, sending a full tablet report...
I20260924 11:52:04.292689 19472 ts_tablet_manager.cc:1428] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260924 11:52:04.294103 19471 ts_tablet_manager.cc:1428] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
W20260924 11:52:04.414909 19186 tablet.cc:2369] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:52:04.417349 19453 tablet.cc:2369] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:52:04.428581 19320 tablet.cc:2369] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:04.538411 19477 raft_consensus.cc:482] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:52:04.538769 19477 raft_consensus.cc:504] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.540448 19477 leader_election.cc:288] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e872c63b2d6745089da757483666516d (127.1.210.3:35037), c21fa3cb898d4b6a95318a0a5f9698c3 (127.1.210.1:40503)
I20260924 11:52:04.547751 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.540875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:37023 (local address 127.1.210.3:35037)
0924 11:52:04.541148 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.541157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.542108 (+ 951us) server_negotiation.cc:407] Connection header received
0924 11:52:04.542349 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.542354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.542419 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.542516 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:04.543176 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.543994 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.546032 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.546787 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.546948 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.547069 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.547368 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.547491 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:52:04.547766 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.541031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:44915 (local address 127.1.210.1:40503)
0924 11:52:04.541291 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.541300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.542131 (+ 831us) server_negotiation.cc:407] Connection header received
0924 11:52:04.542349 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.542354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.542420 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.542519 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:04.543243 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.544103 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.546041 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.546786 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.546943 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.547069 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.547392 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.547511 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 11:52:04.547940 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.540867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:40503 (local address 127.1.210.2:44915)
0924 11:52:04.542018 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:04.542032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:04.542209 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:04.542531 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:04.542535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:04.542586 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:04.543089 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.543097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.544250 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.544255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.545880 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.545892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.547062 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.547068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.547232 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.547254 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:52:04.547363 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 11:52:04.547490 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":73,"spinlock_wait_cycles":52608,"thread_start_us":826,"threads_started":1}
I20260924 11:52:04.548204 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.540832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:35037 (local address 127.1.210.2:37023)
0924 11:52:04.541993 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:04.542012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:04.542207 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:04.542531 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:04.542535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:04.542575 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:04.543031 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.543040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.544137 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.544146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.545872 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:04.545884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.547035 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.547045 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:04.547200 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.547232 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:52:04.547342 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 11:52:04.547473 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":19,"spinlock_wait_cycles":50688,"thread_start_us":752,"threads_started":1}
I20260924 11:52:04.549242 19407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86004f16e61a4c498b6eba230af9656f" candidate_uuid: "e668e655fd694dc19fbd668916f2cd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e872c63b2d6745089da757483666516d" is_pre_election: true
I20260924 11:52:04.549240 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86004f16e61a4c498b6eba230af9656f" candidate_uuid: "e668e655fd694dc19fbd668916f2cd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" is_pre_election: true
I20260924 11:52:04.549825 19140 raft_consensus.cc:2454] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e668e655fd694dc19fbd668916f2cd23 in term 0.
I20260924 11:52:04.549825 19407 raft_consensus.cc:2454] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e668e655fd694dc19fbd668916f2cd23 in term 0.
I20260924 11:52:04.550544 19207 leader_election.cc:302] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c21fa3cb898d4b6a95318a0a5f9698c3, e668e655fd694dc19fbd668916f2cd23; no voters:
I20260924 11:52:04.550886 19477 raft_consensus.cc:2781] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:52:04.551036 19477 raft_consensus.cc:482] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:52:04.551163 19477 raft_consensus.cc:3037] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:04.553377 19477 raft_consensus.cc:504] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.554006 19477 leader_election.cc:288] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [CANDIDATE]: Term 1 election: Requested vote from peers e872c63b2d6745089da757483666516d (127.1.210.3:35037), c21fa3cb898d4b6a95318a0a5f9698c3 (127.1.210.1:40503)
I20260924 11:52:04.554453 19407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86004f16e61a4c498b6eba230af9656f" candidate_uuid: "e668e655fd694dc19fbd668916f2cd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e872c63b2d6745089da757483666516d"
I20260924 11:52:04.554602 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86004f16e61a4c498b6eba230af9656f" candidate_uuid: "e668e655fd694dc19fbd668916f2cd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3"
I20260924 11:52:04.554651 19407 raft_consensus.cc:3037] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:04.554811 19140 raft_consensus.cc:3037] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:04.556928 19407 raft_consensus.cc:2454] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e668e655fd694dc19fbd668916f2cd23 in term 1.
I20260924 11:52:04.557189 19140 raft_consensus.cc:2454] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e668e655fd694dc19fbd668916f2cd23 in term 1.
I20260924 11:52:04.557337 19210 leader_election.cc:302] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e668e655fd694dc19fbd668916f2cd23, e872c63b2d6745089da757483666516d; no voters:
I20260924 11:52:04.557648 19477 raft_consensus.cc:2781] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:04.558100 19477 raft_consensus.cc:686] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 1 LEADER]: Becoming Leader. State: Replica: e668e655fd694dc19fbd668916f2cd23, State: Running, Role: LEADER
I20260924 11:52:04.558560 19477 consensus_queue.cc:237] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } }
I20260924 11:52:04.562958 18996 catalog_manager.cc:5495] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 reported cstate change: term changed from 0 to 1, leader changed from <none> to e668e655fd694dc19fbd668916f2cd23 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "e668e655fd694dc19fbd668916f2cd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e668e655fd694dc19fbd668916f2cd23" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 33525 } health_report { overall_health: HEALTHY } } }
I20260924 11:52:04.613952 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.608349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.1.210.2:33525)
0924 11:52:04.608662 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.608669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.608688 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:52:04.608828 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.608833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.608919 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.609106 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:52:04.609473 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.610187 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.611673 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.612091 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.612238 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.612753 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:52:04.613340 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:52:04.613496 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.613602 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.613686 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260924 11:52:04.622269 19407 raft_consensus.cc:1261] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 1 FOLLOWER]: Refusing update from remote peer e668e655fd694dc19fbd668916f2cd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:04.622269 19140 raft_consensus.cc:1261] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 1 FOLLOWER]: Refusing update from remote peer e668e655fd694dc19fbd668916f2cd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:04.623255 19481 consensus_queue.cc:1035] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c21fa3cb898d4b6a95318a0a5f9698c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:04.623572 19477 consensus_queue.cc:1035] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e872c63b2d6745089da757483666516d" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:04.633481 19484 mvcc.cc:205] Tried to move back new op lower bound from 7332866968042491904 to 7332866967795208192. Current Snapshot: MvccSnapshot[applied={T|T < 7332866968042491904}]
I20260924 11:52:04.648375 18996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:04.648674 18996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:04.652498 18996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57b267a25d545a795fb17f48b42162e: Sending DeleteTablet for 3 replicas of tablet 86004f16e61a4c498b6eba230af9656f
I20260924 11:52:04.653779 19120 tablet_service.cc:1514] Processing DeleteTablet for tablet 86004f16e61a4c498b6eba230af9656f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:04 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:44650
I20260924 11:52:04.653791 19254 tablet_service.cc:1514] Processing DeleteTablet for tablet 86004f16e61a4c498b6eba230af9656f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:04 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:56046
I20260924 11:52:04.653837 19387 tablet_service.cc:1514] Processing DeleteTablet for tablet 86004f16e61a4c498b6eba230af9656f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:04 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:47516
I20260924 11:52:04.656602 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19057
I20260924 11:52:04.663635 19495 tablet_replica.cc:330] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: stopping tablet replica
I20260924 11:52:04.664104 19495 raft_consensus.cc:2229] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:04.664664 19495 raft_consensus.cc:2258] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:04.665555 19493 tablet_replica.cc:330] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: stopping tablet replica
I20260924 11:52:04.665968 19493 raft_consensus.cc:2229] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:52:04.668756 19493 raft_consensus.cc:2258] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:04.669417 19494 tablet_replica.cc:330] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: stopping tablet replica
I20260924 11:52:04.669726 19495 ts_tablet_manager.cc:1905] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:04.673275 19494 raft_consensus.cc:2229] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:04.673810 19494 raft_consensus.cc:2258] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:04.674722 19493 ts_tablet_manager.cc:1905] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:52:04 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:52:04.679510 19495 ts_tablet_manager.cc:1918] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:04.679806 19495 log.cc:1196] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/86004f16e61a4c498b6eba230af9656f
I20260924 11:52:04.680441 19495 ts_tablet_manager.cc:1939] T 86004f16e61a4c498b6eba230af9656f P e872c63b2d6745089da757483666516d: Deleting consensus metadata
I20260924 11:52:04.682534 18983 catalog_manager.cc:4841] TS e872c63b2d6745089da757483666516d (127.1.210.3:35037): tablet 86004f16e61a4c498b6eba230af9656f (table test-table [id=a48899e7aadc42608e5f0de91b995add]) successfully deleted
I20260924 11:52:04.684921 19494 ts_tablet_manager.cc:1905] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:04.693033 19494 ts_tablet_manager.cc:1918] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:04.693357 19494 log.cc:1196] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/86004f16e61a4c498b6eba230af9656f
I20260924 11:52:04.693970 19494 ts_tablet_manager.cc:1939] T 86004f16e61a4c498b6eba230af9656f P c21fa3cb898d4b6a95318a0a5f9698c3: Deleting consensus metadata
I20260924 11:52:04.695828 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.658261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.1.210.1:40503)
0924 11:52:04.658487 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.658497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.665575 (+ 7078us) server_negotiation.cc:407] Connection header received
0924 11:52:04.665767 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.665772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.665881 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.665995 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:04.668224 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.669515 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.671053 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.671469 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.671669 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.687274 (+ 15605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:04.687290 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:04.687308 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:04.687360 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:04.691449 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.692437 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.692446 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.692450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.692616 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.693042 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.693047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.693051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.693565 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:04.693886 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.695156 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.695402 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:52:04.696038 18980 catalog_manager.cc:4841] TS c21fa3cb898d4b6a95318a0a5f9698c3 (127.1.210.1:40503): tablet 86004f16e61a4c498b6eba230af9656f (table test-table [id=a48899e7aadc42608e5f0de91b995add]) successfully deleted
I20260924 11:52:04.696723 19180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:04.696760 19493 ts_tablet_manager.cc:1918] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:04.696940 19493 log.cc:1196] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/86004f16e61a4c498b6eba230af9656f
I20260924 11:52:04.697608 19493 ts_tablet_manager.cc:1939] T 86004f16e61a4c498b6eba230af9656f P e668e655fd694dc19fbd668916f2cd23: Deleting consensus metadata
I20260924 11:52:04.699465 18982 catalog_manager.cc:4841] TS e668e655fd694dc19fbd668916f2cd23 (127.1.210.2:33525): tablet 86004f16e61a4c498b6eba230af9656f (table test-table [id=a48899e7aadc42608e5f0de91b995add]) successfully deleted
I20260924 11:52:04.801527 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19057
I20260924 11:52:04.820170 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19191
Sep 24 11:52:04 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:52:04.836447 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.821221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.1.210.2:33525)
0924 11:52:04.821440 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.821448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.821468 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:52:04.821536 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.821543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.821625 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.821777 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:04.823050 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.823886 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.825430 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.825863 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.826019 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.830354 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:04.830369 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:04.830386 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:04.830427 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:04.833537 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.834246 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.834253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.834255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.834392 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.834808 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.834812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.834814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.835172 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:04.835524 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.835813 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.836049 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:52:04.837360 19314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:04.956581 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19191
I20260924 11:52:04.974802 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19324
Sep 24 11:52:04 dist-test-slave-dd32 krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:52:04.989859 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:04.975528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.1.210.3:35037)
0924 11:52:04.975731 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:04.975740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:04.975761 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:52:04.975962 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:04.975967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:04.976044 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:04.976162 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:04.977528 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.978275 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.979546 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:04.979928 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:04.980072 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:04.984380 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:04.984393 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:04.984408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:04.984440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:04.986983 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.987678 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.987688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.987696 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.987836 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:04.988228 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:04.988233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:04.988236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:04.988594 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:04.988926 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:04.989295 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 11:52:04.989540 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:52:04.990485 19447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:05.078565 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19324
I20260924 11:52:05.094897 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 18964
I20260924 11:52:05.096045 19026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:05.208163 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 18964
2026-09-24T11:52:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4544 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T11:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:05Z Disabled control of system clock
I20260924 11:52:05.282262 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42555
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:52:05.612097 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:05.612594 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:05.612706 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:05.619933 19504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:05.620040 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:05.620090 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:05.620146 19504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:05.620203 19504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:05.629698 19504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:42555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:05.630952 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:05.632876 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:05.645398 19512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:05.645334 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:05.645434 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:05.646241 19504 server_base.cc:1034] running on GCE node
I20260924 11:52:05.647027 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:05.648326 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:05.649559 19504 hybrid_clock.cc:648] HybridClock initialized: now 1790250725649517 us; error 54 us; skew 500 ppm
I20260924 11:52:05.649901 19504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:05.652088 19504 webserver.cc:460] Webserver started at http://127.1.210.62:40541/ using document root <none> and password file <none>
I20260924 11:52:05.652765 19504 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:05.652928 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:05.653308 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:05.656100 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "2fdad1c913d64d35809d3965f0bdcd97"
format_stamp: "Formatted at 2026-09-24 11:52:05 on dist-test-slave-dd32"
server_key: "18d115435b1f3c067826b214f3f262c0"
server_key_iv: "93d846e4710b4de2037e76bcd997cb51"
server_key_version: "encryptionkey@0"
I20260924 11:52:05.656836 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "2fdad1c913d64d35809d3965f0bdcd97"
format_stamp: "Formatted at 2026-09-24 11:52:05 on dist-test-slave-dd32"
server_key: "18d115435b1f3c067826b214f3f262c0"
server_key_iv: "93d846e4710b4de2037e76bcd997cb51"
server_key_version: "encryptionkey@0"
I20260924 11:52:05.661754 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:52:05.665203 19518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:05.666491 19504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:52:05.666702 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "2fdad1c913d64d35809d3965f0bdcd97"
format_stamp: "Formatted at 2026-09-24 11:52:05 on dist-test-slave-dd32"
server_key: "18d115435b1f3c067826b214f3f262c0"
server_key_iv: "93d846e4710b4de2037e76bcd997cb51"
server_key_version: "encryptionkey@0"
I20260924 11:52:05.666925 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:05.697764 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:05.698738 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:05.699046 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:05.717589 19504 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:42555
I20260924 11:52:05.717609 19569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:42555 every 8 connection(s)
I20260924 11:52:05.718989 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:05.722777 19570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:05.724364 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19504
I20260924 11:52:05.724546 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:05.725008 1864 external_mini_cluster.cc:1428] Setting key 32fb3f697135162c520c983ed9d848ea
I20260924 11:52:05.735297 19570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97: Bootstrap starting.
I20260924 11:52:05.738135 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:05.726901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.1.210.62:42555)
0924 11:52:05.727882 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:05.727896 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:05.727937 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:52:05.729561 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:05.729584 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:05.730040 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:05.730432 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:05.730977 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:05.732213 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:05.733805 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:05.734475 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:05.734677 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:05.735936 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:05.735944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:05.735957 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:05.736477 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:05.736559 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:05.736580 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:05.736717 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:05.736935 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 11:52:05.737265 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":356,"threads_started":1}
I20260924 11:52:05.738667 19570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:05.739864 19570 log.cc:824] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:05.742794 19570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97: No bootstrap required, opened a new log
I20260924 11:52:05.750391 19570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdad1c913d64d35809d3965f0bdcd97" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42555 } }
I20260924 11:52:05.750867 19570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:05.750993 19570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdad1c913d64d35809d3965f0bdcd97, State: Initialized, Role: FOLLOWER
I20260924 11:52:05.751583 19570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdad1c913d64d35809d3965f0bdcd97" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42555 } }
I20260924 11:52:05.751837 19570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:05.752000 19570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:05.752175 19570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:05.754657 19570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdad1c913d64d35809d3965f0bdcd97" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42555 } }
I20260924 11:52:05.755227 19570 leader_election.cc:302] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fdad1c913d64d35809d3965f0bdcd97; no voters:
I20260924 11:52:05.755805 19570 leader_election.cc:288] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:05.755944 19575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:05.756659 19575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [term 1 LEADER]: Becoming Leader. State: Replica: 2fdad1c913d64d35809d3965f0bdcd97, State: Running, Role: LEADER
I20260924 11:52:05.757210 19575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdad1c913d64d35809d3965f0bdcd97" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42555 } }
I20260924 11:52:05.758003 19570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:05.760144 19577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fdad1c913d64d35809d3965f0bdcd97. Latest consensus state: current_term: 1 leader_uuid: "2fdad1c913d64d35809d3965f0bdcd97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdad1c913d64d35809d3965f0bdcd97" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42555 } } }
I20260924 11:52:05.760389 19576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fdad1c913d64d35809d3965f0bdcd97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdad1c913d64d35809d3965f0bdcd97" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42555 } } }
I20260924 11:52:05.760660 19577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:05.760681 19576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:05.762324 19581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:05.767627 19581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:05.776852 19581 catalog_manager.cc:1347] Generated new cluster ID: fa696fa4c0dc4ebda4ebaef1a62ce5b4
I20260924 11:52:05.776965 19581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:05.813863 19581 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:05.815166 19581 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:05.828419 19581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97: Generated new TSK 0
I20260924 11:52:05.829368 19581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:05.843573 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42555
--builtin_ntp_servers=127.1.210.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:52:06.174221 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:06.174556 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:06.174717 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:06.181856 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:06.181994 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:06.182176 19594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:52:06.191329 19594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:06.192600 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:06.194367 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:06.207252 19599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:06.207341 19600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:06.207275 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:06.212900 19594 server_base.cc:1034] running on GCE node
I20260924 11:52:06.213590 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:06.214556 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:06.215804 19594 hybrid_clock.cc:648] HybridClock initialized: now 1790250726215767 us; error 56 us; skew 500 ppm
I20260924 11:52:06.216122 19594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:06.218413 19594 webserver.cc:460] Webserver started at http://127.1.210.1:41293/ using document root <none> and password file <none>
I20260924 11:52:06.219115 19594 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:06.219252 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:06.219579 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:06.222524 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "7ae4591541664956aedf2f1f0ee962c3"
format_stamp: "Formatted at 2026-09-24 11:52:06 on dist-test-slave-dd32"
server_key: "5194bc073a3263182d4cba24870ea813"
server_key_iv: "614000aa4e597d0d906cfc08d5899a53"
server_key_version: "encryptionkey@0"
I20260924 11:52:06.223315 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "7ae4591541664956aedf2f1f0ee962c3"
format_stamp: "Formatted at 2026-09-24 11:52:06 on dist-test-slave-dd32"
server_key: "5194bc073a3263182d4cba24870ea813"
server_key_iv: "614000aa4e597d0d906cfc08d5899a53"
server_key_version: "encryptionkey@0"
I20260924 11:52:06.228448 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:52:06.231881 19608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:06.233188 19594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:06.233350 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "7ae4591541664956aedf2f1f0ee962c3"
format_stamp: "Formatted at 2026-09-24 11:52:06 on dist-test-slave-dd32"
server_key: "5194bc073a3263182d4cba24870ea813"
server_key_iv: "614000aa4e597d0d906cfc08d5899a53"
server_key_version: "encryptionkey@0"
I20260924 11:52:06.233556 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:06.259759 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:06.260576 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:06.260860 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:06.261909 19594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:06.263394 19594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:06.263481 19594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:06.263665 19594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:06.263758 19594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:06.296890 19594 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:42631
I20260924 11:52:06.296981 19720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:42631 every 8 connection(s)
I20260924 11:52:06.298259 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:52:06.302907 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19594
I20260924 11:52:06.303061 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:52:06.303510 1864 external_mini_cluster.cc:1428] Setting key 7bbe962d101849320766900ead248239
I20260924 11:52:06.306164 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42555
--builtin_ntp_servers=127.1.210.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:52:06.311626 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:06.300827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:46743 (local address 127.1.210.62:42555)
0924 11:52:06.301322 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:06.301328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:06.302531 (+ 1203us) server_negotiation.cc:407] Connection header received
0924 11:52:06.304612 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:06.304617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:06.304701 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:06.304830 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:06.306050 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.306893 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.308819 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.309263 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.309399 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:06.310805 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:06.310810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:06.310814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:06.310832 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:06.310870 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:06.310882 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:06.310978 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:06.311208 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 11:52:06.311304 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260924 11:52:06.312606 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:06.301485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42555 (local address 127.1.210.1:46743)
0924 11:52:06.302372 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:06.302408 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:06.304344 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:06.305112 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:06.305127 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:06.305257 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:06.305880 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:06.305896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.307067 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.307072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:06.308672 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:06.308684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.309488 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.309497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:06.309760 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:06.310469 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:06.310472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:06.310510 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:06.310596 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:06.310978 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:06.310991 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:06.311002 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:52:06.311197 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 11:52:06.311590 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":273,"threads_started":1}
I20260924 11:52:06.314311 19721 heartbeater.cc:347] Connected to a master server at 127.1.210.62:42555
I20260924 11:52:06.314623 19721 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:06.315335 19721 heartbeater.cc:511] Master 127.1.210.62:42555 requested a full tablet report, sending...
I20260924 11:52:06.317482 19535 ts_manager.cc:194] Registered new tserver with Master: 7ae4591541664956aedf2f1f0ee962c3 (127.1.210.1:42631)
I20260924 11:52:06.318969 19535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.1:46743
W20260924 11:52:06.629667 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:06.629998 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:06.630159 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:06.638295 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:06.638415 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:06.638586 19725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:52:06.647650 19725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:06.648944 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:06.650704 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:06.663987 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:06.664047 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:06.663954 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:06.664777 19725 server_base.cc:1034] running on GCE node
I20260924 11:52:06.665421 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:06.666333 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:06.667562 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790250726667520 us; error 57 us; skew 500 ppm
I20260924 11:52:06.667881 19725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:06.670219 19725 webserver.cc:460] Webserver started at http://127.1.210.2:35897/ using document root <none> and password file <none>
I20260924 11:52:06.670969 19725 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:06.671123 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:06.671440 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:06.674259 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "b4fe62915b3b46699c36e5b46ee6cad8"
format_stamp: "Formatted at 2026-09-24 11:52:06 on dist-test-slave-dd32"
server_key: "0c0012e6c3029f1f985c3087d0861f93"
server_key_iv: "89a42a9774b788e76b6c46abfdd71bd2"
server_key_version: "encryptionkey@0"
I20260924 11:52:06.675050 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "b4fe62915b3b46699c36e5b46ee6cad8"
format_stamp: "Formatted at 2026-09-24 11:52:06 on dist-test-slave-dd32"
server_key: "0c0012e6c3029f1f985c3087d0861f93"
server_key_iv: "89a42a9774b788e76b6c46abfdd71bd2"
server_key_version: "encryptionkey@0"
I20260924 11:52:06.680034 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:52:06.683557 19739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:06.684881 19725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:52:06.685045 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "b4fe62915b3b46699c36e5b46ee6cad8"
format_stamp: "Formatted at 2026-09-24 11:52:06 on dist-test-slave-dd32"
server_key: "0c0012e6c3029f1f985c3087d0861f93"
server_key_iv: "89a42a9774b788e76b6c46abfdd71bd2"
server_key_version: "encryptionkey@0"
I20260924 11:52:06.685317 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:06.707362 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:06.708209 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:06.708506 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:06.709538 19725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:06.711019 19725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:06.711109 19725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:06.711267 19725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:06.711359 19725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:06.745815 19725 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:39379
I20260924 11:52:06.745913 19851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:39379 every 8 connection(s)
I20260924 11:52:06.747103 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:52:06.751088 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19725
I20260924 11:52:06.751271 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:52:06.751718 1864 external_mini_cluster.cc:1428] Setting key 262a38cce928b535b2761aadfaac35b9
I20260924 11:52:06.754688 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42555
--builtin_ntp_servers=127.1.210.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:52:06.760052 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:06.749645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:57695 (local address 127.1.210.62:42555)
0924 11:52:06.749817 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:06.749821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:06.751171 (+ 1350us) server_negotiation.cc:407] Connection header received
0924 11:52:06.753377 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:06.753381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:06.753444 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:06.753572 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:06.754737 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.755542 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.757141 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.757540 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.757682 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:06.759226 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:06.759237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:06.759241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:06.759261 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:06.759291 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:06.759303 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:06.759398 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:06.759643 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 11:52:06.759741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:52:06.761153 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:06.750072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42555 (local address 127.1.210.2:57695)
0924 11:52:06.751014 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:06.751050 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:06.753047 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:06.753858 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:06.753872 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:06.753986 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:06.754548 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:06.754563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.755707 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.755713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:06.756956 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:06.756968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:06.757773 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:06.757780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:06.758071 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:06.758865 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:06.758867 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:06.758908 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:06.758991 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:06.759398 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:06.759411 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:06.759421 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:52:06.759634 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 11:52:06.760036 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260924 11:52:06.762806 19852 heartbeater.cc:347] Connected to a master server at 127.1.210.62:42555
I20260924 11:52:06.763276 19852 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:06.764460 19852 heartbeater.cc:511] Master 127.1.210.62:42555 requested a full tablet report, sending...
I20260924 11:52:06.766536 19535 ts_manager.cc:194] Registered new tserver with Master: b4fe62915b3b46699c36e5b46ee6cad8 (127.1.210.2:39379)
I20260924 11:52:06.767812 19535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.2:57695
W20260924 11:52:07.088439 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:07.088774 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:07.088863 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:07.096221 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:07.096328 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:07.096437 19855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:52:07.105628 19855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:07.106892 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:07.108574 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:07.121555 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:07.121680 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:07.122089 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:07.123453 19855 server_base.cc:1034] running on GCE node
I20260924 11:52:07.124226 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:07.125253 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:07.126444 19855 hybrid_clock.cc:648] HybridClock initialized: now 1790250727126374 us; error 58 us; skew 500 ppm
I20260924 11:52:07.126804 19855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:07.128991 19855 webserver.cc:460] Webserver started at http://127.1.210.3:46479/ using document root <none> and password file <none>
I20260924 11:52:07.129750 19855 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:07.129873 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:07.130213 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:07.133138 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "01af25dbc9a34e9591d09cc20a7873b4"
format_stamp: "Formatted at 2026-09-24 11:52:07 on dist-test-slave-dd32"
server_key: "9ca7f2123fad86fce83cf09e7f6cb19d"
server_key_iv: "e6ddda0b1b3bd6bd63ab8205ef9d34be"
server_key_version: "encryptionkey@0"
I20260924 11:52:07.133956 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "01af25dbc9a34e9591d09cc20a7873b4"
format_stamp: "Formatted at 2026-09-24 11:52:07 on dist-test-slave-dd32"
server_key: "9ca7f2123fad86fce83cf09e7f6cb19d"
server_key_iv: "e6ddda0b1b3bd6bd63ab8205ef9d34be"
server_key_version: "encryptionkey@0"
I20260924 11:52:07.139199 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:52:07.142704 19869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:07.143992 19855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:52:07.144189 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "01af25dbc9a34e9591d09cc20a7873b4"
format_stamp: "Formatted at 2026-09-24 11:52:07 on dist-test-slave-dd32"
server_key: "9ca7f2123fad86fce83cf09e7f6cb19d"
server_key_iv: "e6ddda0b1b3bd6bd63ab8205ef9d34be"
server_key_version: "encryptionkey@0"
I20260924 11:52:07.144408 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:07.161805 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:07.162658 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:07.162981 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:07.164098 19855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:07.165603 19855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:07.165724 19855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:07.165855 19855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:07.165946 19855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:07.199124 19855 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:43349
I20260924 11:52:07.199174 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:43349 every 8 connection(s)
I20260924 11:52:07.200476 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:52:07.202351 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 19855
I20260924 11:52:07.202553 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:52:07.203042 1864 external_mini_cluster.cc:1428] Setting key b68dd8381587acd6c216dab455469bb7
I20260924 11:52:07.212828 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.203181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:55717 (local address 127.1.210.62:42555)
0924 11:52:07.203402 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.203409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.205096 (+ 1687us) server_negotiation.cc:407] Connection header received
0924 11:52:07.206870 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.206874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.206939 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.207019 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:07.208029 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.208746 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.210312 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.210736 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.210830 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.212161 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:07.212166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:07.212169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:07.212186 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:07.212212 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:07.212223 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:07.212288 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.212486 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.212596 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:52:07.213814 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.203802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:42555 (local address 127.1.210.3:55717)
0924 11:52:07.204937 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:07.204987 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:07.206632 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:07.207234 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:07.207242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:07.207335 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:07.207891 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.207908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.208877 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.208880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.210189 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.210201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.210925 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.210930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.211145 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.211879 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:07.211882 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:07.211919 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:07.212000 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:07.212306 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:07.212313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:07.212319 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:52:07.212457 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 11:52:07.212748 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":390,"threads_started":1}
I20260924 11:52:07.215171 19982 heartbeater.cc:347] Connected to a master server at 127.1.210.62:42555
I20260924 11:52:07.215463 19982 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:07.216200 19982 heartbeater.cc:511] Master 127.1.210.62:42555 requested a full tablet report, sending...
I20260924 11:52:07.217916 19535 ts_manager.cc:194] Registered new tserver with Master: 01af25dbc9a34e9591d09cc20a7873b4 (127.1.210.3:43349)
I20260924 11:52:07.218487 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:52:07.218868 19535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.3:55717
I20260924 11:52:07.228477 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.223063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.1.210.62:42555)
0924 11:52:07.223234 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.223238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.223673 (+ 435us) server_negotiation.cc:407] Connection header received
0924 11:52:07.223804 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.223808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.223867 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.223948 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:07.224391 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.225140 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.226479 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.226932 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.227022 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.227948 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:07.227953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:07.227956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:07.227972 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:07.228002 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:07.228013 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:07.228080 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.228201 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.228268 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:52:07.232084 19535 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:52:07.238941 19535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:52:07.264400 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.251374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.1.210.3:43349)
0924 11:52:07.252455 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.252466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.252984 (+ 518us) server_negotiation.cc:407] Connection header received
0924 11:52:07.253341 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.253352 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.253650 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.253834 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:07.255156 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.257275 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.261133 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.262944 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.263064 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.263397 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.263818 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.263983 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":732,"threads_started":1}
I20260924 11:52:07.264375 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.250727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43349 (local address 127.0.0.1:38172)
0924 11:52:07.252885 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:07.252904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:07.253109 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:07.254319 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:07.254347 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:07.254388 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:07.254873 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.254890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.257518 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.257527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.260928 (+ 3401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.260938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.263191 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.263198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.263553 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.263612 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 11:52:07.263755 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 11:52:07.263948 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2028,"mutex_wait_us":90,"thread_start_us":1909,"threads_started":1}
I20260924 11:52:07.266116 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.250490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:42631 (local address 127.0.0.1:42248)
0924 11:52:07.251293 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:07.251351 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:07.251554 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:07.259155 (+ 7601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:07.259159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:07.259183 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:07.259654 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.259670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.261301 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.261307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.263375 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.263393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.265300 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.265309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.265480 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.265513 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:52:07.265596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 11:52:07.265755 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":1,"thread_start_us":297,"threads_started":1}
I20260924 11:52:07.266999 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.250773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.1.210.1:42631)
0924 11:52:07.258172 (+ 7399us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.258195 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.258230 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:52:07.258349 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.258360 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.258643 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.258891 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:07.259839 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.261166 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.263584 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.265108 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.265237 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.265469 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.266484 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.266654 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7171,"thread_start_us":7033,"threads_started":1}
I20260924 11:52:07.268652 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.253024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:39379 (local address 127.0.0.1:39290)
0924 11:52:07.258888 (+ 5864us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:07.258906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:07.259028 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:07.260026 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:07.260030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:07.260054 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:07.260438 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.260448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.262295 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.262303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.265200 (+ 2897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.265216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.267072 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.267086 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.267251 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.268136 (+ 885us) client_negotiation.cc:769] Sending connection context
0924 11:52:07.268233 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:52:07.268353 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5740,"mutex_wait_us":21,"thread_start_us":5522,"threads_started":1}
I20260924 11:52:07.268793 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.253502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.1.210.2:39379)
0924 11:52:07.258472 (+ 4970us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.258484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.259155 (+ 671us) server_negotiation.cc:407] Connection header received
0924 11:52:07.259259 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.259267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.259546 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.259736 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:07.260653 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.261843 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.265418 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.266878 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.267029 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.267259 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.268301 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.268472 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4811,"thread_start_us":4717,"threads_started":1}
I20260924 11:52:07.268909 19917 tablet_service.cc:1467] Processing CreateTablet for tablet 91691a0bdc974aa19c474b115a8b7fad (DEFAULT_TABLE table=test-table [id=e8128bbdfd1245729eecb5c0f5f8ac0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:07.270521 19656 tablet_service.cc:1467] Processing CreateTablet for tablet 91691a0bdc974aa19c474b115a8b7fad (DEFAULT_TABLE table=test-table [id=e8128bbdfd1245729eecb5c0f5f8ac0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:07.271224 19917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91691a0bdc974aa19c474b115a8b7fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:07.271778 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91691a0bdc974aa19c474b115a8b7fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:07.272152 19787 tablet_service.cc:1467] Processing CreateTablet for tablet 91691a0bdc974aa19c474b115a8b7fad (DEFAULT_TABLE table=test-table [id=e8128bbdfd1245729eecb5c0f5f8ac0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:07.273429 19787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91691a0bdc974aa19c474b115a8b7fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:07.283093 20000 tablet_bootstrap.cc:492] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Bootstrap starting.
I20260924 11:52:07.287047 20000 tablet_bootstrap.cc:654] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:07.288007 20001 tablet_bootstrap.cc:492] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Bootstrap starting.
I20260924 11:52:07.288482 20000 log.cc:824] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:07.289736 20002 tablet_bootstrap.cc:492] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Bootstrap starting.
I20260924 11:52:07.292465 20001 tablet_bootstrap.cc:654] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:07.293052 20002 tablet_bootstrap.cc:654] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:07.294070 20002 log.cc:824] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:07.294103 20001 log.cc:824] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:07.297168 20000 tablet_bootstrap.cc:492] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: No bootstrap required, opened a new log
I20260924 11:52:07.297253 20002 tablet_bootstrap.cc:492] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: No bootstrap required, opened a new log
I20260924 11:52:07.297461 20000 ts_tablet_manager.cc:1397] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260924 11:52:07.297499 20002 ts_tablet_manager.cc:1397] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 11:52:07.297602 20001 tablet_bootstrap.cc:492] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: No bootstrap required, opened a new log
I20260924 11:52:07.297828 20001 ts_tablet_manager.cc:1397] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260924 11:52:07.308266 20001 raft_consensus.cc:348] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.308266 20002 raft_consensus.cc:348] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.308722 20002 raft_consensus.cc:374] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:07.308765 20001 raft_consensus.cc:374] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:07.308858 20001 raft_consensus.cc:729] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01af25dbc9a34e9591d09cc20a7873b4, State: Initialized, Role: FOLLOWER
I20260924 11:52:07.308880 20002 raft_consensus.cc:729] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fe62915b3b46699c36e5b46ee6cad8, State: Initialized, Role: FOLLOWER
I20260924 11:52:07.309012 20000 raft_consensus.cc:348] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.309700 20000 raft_consensus.cc:374] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:07.309566 20001 consensus_queue.cc:260] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.309566 20002 consensus_queue.cc:260] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.309836 20000 raft_consensus.cc:729] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae4591541664956aedf2f1f0ee962c3, State: Initialized, Role: FOLLOWER
I20260924 11:52:07.310473 20000 consensus_queue.cc:260] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.317767 19852 heartbeater.cc:503] Master 127.1.210.62:42555 was elected leader, sending a full tablet report...
I20260924 11:52:07.318511 20000 ts_tablet_manager.cc:1428] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Time spent starting tablet: real 0.021s user 0.018s sys 0.001s
I20260924 11:52:07.318562 20002 ts_tablet_manager.cc:1428] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20260924 11:52:07.318885 19721 heartbeater.cc:503] Master 127.1.210.62:42555 was elected leader, sending a full tablet report...
I20260924 11:52:07.319002 19982 heartbeater.cc:503] Master 127.1.210.62:42555 was elected leader, sending a full tablet report...
I20260924 11:52:07.320120 20001 ts_tablet_manager.cc:1428] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Time spent starting tablet: real 0.022s user 0.011s sys 0.010s
I20260924 11:52:07.448387 20006 raft_consensus.cc:482] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:52:07.448701 20006 raft_consensus.cc:504] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.450484 20006 leader_election.cc:288] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ae4591541664956aedf2f1f0ee962c3 (127.1.210.1:42631), 01af25dbc9a34e9591d09cc20a7873b4 (127.1.210.3:43349)
W20260924 11:52:07.452240 19983 tablet.cc:2369] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:07.457891 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.450825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:54053 (local address 127.1.210.1:42631)
0924 11:52:07.451063 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.451073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.451801 (+ 728us) server_negotiation.cc:407] Connection header received
0924 11:52:07.452148 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.452159 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.452255 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.452327 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:07.453378 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.454067 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.456148 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.456888 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.457043 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.457167 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.457511 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.457638 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:52:07.457906 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.450975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:45071 (local address 127.1.210.3:43349)
0924 11:52:07.451210 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.451218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.451877 (+ 659us) server_negotiation.cc:407] Connection header received
0924 11:52:07.452143 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.452150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.452233 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.452303 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:07.453319 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.454022 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.456169 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.456935 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.457105 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.457204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.457537 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.457660 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:52:07.458367 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.450785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:43349 (local address 127.1.210.2:45071)
0924 11:52:07.451727 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:07.451773 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:07.451947 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:07.452495 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:07.452499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:07.452529 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:07.453146 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.453161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.454162 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.454170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.455952 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.455964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.457150 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.457156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.457376 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.457398 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:52:07.457500 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:52:07.457672 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":187,"spinlock_wait_cycles":1792,"thread_start_us":658,"threads_started":1}
I20260924 11:52:07.458384 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.450744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:42631 (local address 127.1.210.2:54053)
0924 11:52:07.451682 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:07.451701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:07.451959 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:07.452521 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:07.452525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:07.452547 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:07.453233 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.453246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.454232 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.454240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.455942 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:07.455954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.457115 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.457122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:07.457302 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.457373 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 11:52:07.457484 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 11:52:07.457615 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":125,"spinlock_wait_cycles":21632,"thread_start_us":558,"threads_started":1}
I20260924 11:52:07.459426 19937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91691a0bdc974aa19c474b115a8b7fad" candidate_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01af25dbc9a34e9591d09cc20a7873b4" is_pre_election: true
I20260924 11:52:07.459681 19676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91691a0bdc974aa19c474b115a8b7fad" candidate_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ae4591541664956aedf2f1f0ee962c3" is_pre_election: true
I20260924 11:52:07.459935 19937 raft_consensus.cc:2454] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fe62915b3b46699c36e5b46ee6cad8 in term 0.
I20260924 11:52:07.460229 19676 raft_consensus.cc:2454] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fe62915b3b46699c36e5b46ee6cad8 in term 0.
I20260924 11:52:07.460618 19741 leader_election.cc:302] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01af25dbc9a34e9591d09cc20a7873b4, b4fe62915b3b46699c36e5b46ee6cad8; no voters:
I20260924 11:52:07.461048 20006 raft_consensus.cc:2781] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:52:07.461249 20006 raft_consensus.cc:482] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:52:07.461374 20006 raft_consensus.cc:3037] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:07.463641 20006 raft_consensus.cc:504] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.464289 20006 leader_election.cc:288] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [CANDIDATE]: Term 1 election: Requested vote from peers 7ae4591541664956aedf2f1f0ee962c3 (127.1.210.1:42631), 01af25dbc9a34e9591d09cc20a7873b4 (127.1.210.3:43349)
I20260924 11:52:07.464793 19676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91691a0bdc974aa19c474b115a8b7fad" candidate_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ae4591541664956aedf2f1f0ee962c3"
I20260924 11:52:07.464995 19676 raft_consensus.cc:3037] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:07.467236 19676 raft_consensus.cc:2454] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fe62915b3b46699c36e5b46ee6cad8 in term 1.
I20260924 11:52:07.467586 19937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91691a0bdc974aa19c474b115a8b7fad" candidate_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01af25dbc9a34e9591d09cc20a7873b4"
I20260924 11:52:07.467648 19742 leader_election.cc:302] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ae4591541664956aedf2f1f0ee962c3, b4fe62915b3b46699c36e5b46ee6cad8; no voters:
I20260924 11:52:07.467819 19937 raft_consensus.cc:3037] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:07.467983 20006 raft_consensus.cc:2781] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:07.468515 20006 raft_consensus.cc:686] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 1 LEADER]: Becoming Leader. State: Replica: b4fe62915b3b46699c36e5b46ee6cad8, State: Running, Role: LEADER
I20260924 11:52:07.469018 20006 consensus_queue.cc:237] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } }
I20260924 11:52:07.470527 19937 raft_consensus.cc:2454] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fe62915b3b46699c36e5b46ee6cad8 in term 1.
I20260924 11:52:07.474079 19535 catalog_manager.cc:5495] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4fe62915b3b46699c36e5b46ee6cad8 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fe62915b3b46699c36e5b46ee6cad8" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 39379 } health_report { overall_health: HEALTHY } } }
W20260924 11:52:07.499238 19853 tablet.cc:2369] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:07.510004 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.505136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.1.210.2:39379)
0924 11:52:07.505294 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.505298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.505315 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:52:07.505414 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.505417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.505502 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.505637 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:52:07.505930 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.506616 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.508001 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.508408 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.508503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.508953 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:52:07.509538 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:52:07.509661 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.509752 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.509811 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:52:07.518147 19676 raft_consensus.cc:1261] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 1 FOLLOWER]: Refusing update from remote peer b4fe62915b3b46699c36e5b46ee6cad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:07.518147 19937 raft_consensus.cc:1261] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 1 FOLLOWER]: Refusing update from remote peer b4fe62915b3b46699c36e5b46ee6cad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:07.519281 20011 consensus_queue.cc:1035] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ae4591541664956aedf2f1f0ee962c3" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:07.519618 20006 consensus_queue.cc:1035] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01af25dbc9a34e9591d09cc20a7873b4" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:07.533241 20015 mvcc.cc:205] Tried to move back new op lower bound from 7332866979903266816 to 7332866979717320704. Current Snapshot: MvccSnapshot[applied={T|T < 7332866979903266816}]
I20260924 11:52:07.546798 19534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:07.547124 19534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:07.551384 19534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fdad1c913d64d35809d3965f0bdcd97: Sending DeleteTablet for 3 replicas of tablet 91691a0bdc974aa19c474b115a8b7fad
I20260924 11:52:07.552508 19656 tablet_service.cc:1514] Processing DeleteTablet for tablet 91691a0bdc974aa19c474b115a8b7fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:07 UTC) from {username='slave'} at 127.0.0.1:42248
I20260924 11:52:07.552948 19787 tablet_service.cc:1514] Processing DeleteTablet for tablet 91691a0bdc974aa19c474b115a8b7fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:07 UTC) from {username='slave'} at 127.0.0.1:39290
I20260924 11:52:07.553234 19917 tablet_service.cc:1514] Processing DeleteTablet for tablet 91691a0bdc974aa19c474b115a8b7fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:07 UTC) from {username='slave'} at 127.0.0.1:38172
I20260924 11:52:07.554528 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19594
I20260924 11:52:07.558452 20026 tablet_replica.cc:330] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: stopping tablet replica
I20260924 11:52:07.559054 20026 raft_consensus.cc:2229] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:07.563417 20026 raft_consensus.cc:2258] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:07.566286 20024 tablet_replica.cc:330] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: stopping tablet replica
I20260924 11:52:07.566725 20026 ts_tablet_manager.cc:1905] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:07.566895 20024 raft_consensus.cc:2229] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:07.567422 20024 raft_consensus.cc:2258] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:07.568194 20025 tablet_replica.cc:330] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: stopping tablet replica
I20260924 11:52:07.568589 20025 raft_consensus.cc:2229] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:52:07.569271 20025 raft_consensus.cc:2258] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:07.569585 20024 ts_tablet_manager.cc:1905] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:07.571101 20025 ts_tablet_manager.cc:1905] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:07.573244 20026 ts_tablet_manager.cc:1918] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:07.573438 20026 log.cc:1196] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/91691a0bdc974aa19c474b115a8b7fad
I20260924 11:52:07.573940 20026 ts_tablet_manager.cc:1939] T 91691a0bdc974aa19c474b115a8b7fad P 01af25dbc9a34e9591d09cc20a7873b4: Deleting consensus metadata
I20260924 11:52:07.575577 19520 catalog_manager.cc:4841] TS 01af25dbc9a34e9591d09cc20a7873b4 (127.1.210.3:43349): tablet 91691a0bdc974aa19c474b115a8b7fad (table test-table [id=e8128bbdfd1245729eecb5c0f5f8ac0b]) successfully deleted
I20260924 11:52:07.576478 20024 ts_tablet_manager.cc:1918] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:07.576639 20024 log.cc:1196] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/91691a0bdc974aa19c474b115a8b7fad
I20260924 11:52:07.577215 20024 ts_tablet_manager.cc:1939] T 91691a0bdc974aa19c474b115a8b7fad P 7ae4591541664956aedf2f1f0ee962c3: Deleting consensus metadata
I20260924 11:52:07.578116 20025 ts_tablet_manager.cc:1918] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:07.578279 20025 log.cc:1196] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/91691a0bdc974aa19c474b115a8b7fad
I20260924 11:52:07.578848 20025 ts_tablet_manager.cc:1939] T 91691a0bdc974aa19c474b115a8b7fad P b4fe62915b3b46699c36e5b46ee6cad8: Deleting consensus metadata
I20260924 11:52:07.578969 19521 catalog_manager.cc:4841] TS 7ae4591541664956aedf2f1f0ee962c3 (127.1.210.1:42631): tablet 91691a0bdc974aa19c474b115a8b7fad (table test-table [id=e8128bbdfd1245729eecb5c0f5f8ac0b]) successfully deleted
I20260924 11:52:07.580422 19520 catalog_manager.cc:4841] TS b4fe62915b3b46699c36e5b46ee6cad8 (127.1.210.2:39379): tablet 91691a0bdc974aa19c474b115a8b7fad (table test-table [id=e8128bbdfd1245729eecb5c0f5f8ac0b]) successfully deleted
I20260924 11:52:07.581964 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.556395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.1.210.1:42631)
0924 11:52:07.556640 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.556650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.563216 (+ 6566us) server_negotiation.cc:407] Connection header received
0924 11:52:07.563542 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.563552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.563684 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.563798 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:07.564719 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.566197 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.576308 (+ 10111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.577102 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.577269 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.580175 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:07.580187 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:07.580208 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:07.581030 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:07.581119 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:07.581147 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:07.581273 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.581413 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.581536 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:52:07.582662 19716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:07.720463 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19594
I20260924 11:52:07.740412 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19725
I20260924 11:52:07.748287 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.741333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.1.210.2:39379)
0924 11:52:07.741526 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.741532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.741635 (+ 103us) server_negotiation.cc:407] Connection header received
0924 11:52:07.741850 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.741860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.741961 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.742096 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:07.742517 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.743273 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.744604 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.745018 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.745233 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.746632 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:07.746641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:07.746655 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:07.747253 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:07.747299 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:07.747322 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:07.747458 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.747609 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.747776 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:52:07.749334 19847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:07.862545 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19725
I20260924 11:52:07.881556 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19855
I20260924 11:52:07.888587 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:07.882357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.1.210.3:43349)
0924 11:52:07.882538 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:07.882545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:07.882590 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:52:07.882811 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:07.882815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:07.882896 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:07.883055 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:07.883430 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.884166 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.885569 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:07.885955 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:07.886102 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:07.887199 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:07.887208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:07.887223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:07.887790 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:07.887843 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:07.887861 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:07.888020 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:07.888152 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 11:52:07.888266 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:52:07.889348 19977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:07.977466 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19855
I20260924 11:52:07.993633 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 19504
I20260924 11:52:07.994710 19565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:08.103137 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 19504
2026-09-24T11:52:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2897 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:08 dist-test-slave-dd32 krb5kdc[20034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:08 dist-test-slave-dd32 krb5kdc[20034](info): set up 2 sockets
Sep 24 11:52:08 dist-test-slave-dd32 krb5kdc[20034](info): commencing operation
krb5kdc: starting...
W20260924 11:52:09.209147 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:09 dist-test-slave-dd32 krb5kdc[20034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:09.458688 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:34197
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:09.804160 20047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:09.804667 20047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:09.804860 20047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:09.812237 20047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:09.812508 20047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:09.812662 20047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:09.812795 20047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:09.812906 20047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:09.824471 20047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:09.826154 20047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:09.827664 20047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:09.841255 20052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:09.841289 20053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:09.841332 20055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:09.842850 20047 server_base.cc:1034] running on GCE node
I20260924 11:52:09.843752 20047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:09.845337 20047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:09.846580 20047 hybrid_clock.cc:648] HybridClock initialized: now 1790250729846534 us; error 38 us; skew 500 ppm
I20260924 11:52:09.847029 20047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:09 dist-test-slave-dd32 krb5kdc[20034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250729, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:09.854110 20047 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:09.855665 20047 webserver.cc:460] Webserver started at http://127.1.210.62:38547/ using document root <none> and password file <none>
I20260924 11:52:09.856318 20047 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:09.856413 20047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:09.856680 20047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:09.859712 20047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "3618dfd9ec554bec88eefde1811fc3a5"
format_stamp: "Formatted at 2026-09-24 11:52:09 on dist-test-slave-dd32"
server_key: "8a54976dba0cc68974e4c9c2a1daa879"
server_key_iv: "5c6b870f20fe394b3e581ebf590c6d0a"
server_key_version: "encryptionkey@0"
I20260924 11:52:09.860397 20047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "3618dfd9ec554bec88eefde1811fc3a5"
format_stamp: "Formatted at 2026-09-24 11:52:09 on dist-test-slave-dd32"
server_key: "8a54976dba0cc68974e4c9c2a1daa879"
server_key_iv: "5c6b870f20fe394b3e581ebf590c6d0a"
server_key_version: "encryptionkey@0"
I20260924 11:52:09.865504 20047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:52:09.869256 20062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:09.870594 20047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:09.870788 20047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "3618dfd9ec554bec88eefde1811fc3a5"
format_stamp: "Formatted at 2026-09-24 11:52:09 on dist-test-slave-dd32"
server_key: "8a54976dba0cc68974e4c9c2a1daa879"
server_key_iv: "5c6b870f20fe394b3e581ebf590c6d0a"
server_key_version: "encryptionkey@0"
I20260924 11:52:09.870994 20047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:09.891904 20047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:09.896442 20047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:09.896762 20047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:09.915400 20047 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:34197
I20260924 11:52:09.915454 20113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:34197 every 8 connection(s)
I20260924 11:52:09.916842 20047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:09.920559 20114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:09.923265 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20047
I20260924 11:52:09.923511 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:09.923956 1864 external_mini_cluster.cc:1428] Setting key a07ebd479026eca35ecee3e88bf08253
I20260924 11:52:09.933481 20114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5: Bootstrap starting.
I20260924 11:52:09.937044 20114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:09.938102 20114 log.cc:824] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:52:09 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:09.941267 20114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5: No bootstrap required, opened a new log
I20260924 11:52:09.947762 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:09.925953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.1.210.62:34197)
0924 11:52:09.926751 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:09.926765 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:09.926813 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:52:09.929235 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:09.929265 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:09.929793 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:09.930242 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:09.931513 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:09.933495 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:09.934992 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:09.936045 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:09.936240 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:09.940580 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:09.940589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:09.940602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:09.940638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:09.943730 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:09.944574 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:09.944580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:09.944586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:09.944748 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:09.945185 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:09.945188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:09.945190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:09.945692 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:09.945918 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:09.946379 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 11:52:09.946793 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":343,"threads_started":1}
I20260924 11:52:09.950064 20114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3618dfd9ec554bec88eefde1811fc3a5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34197 } }
I20260924 11:52:09.950527 20114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:09.950646 20114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3618dfd9ec554bec88eefde1811fc3a5, State: Initialized, Role: FOLLOWER
I20260924 11:52:09.951200 20114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3618dfd9ec554bec88eefde1811fc3a5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34197 } }
I20260924 11:52:09.951447 20114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:09.951592 20114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:09.951786 20114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:09.953948 20114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3618dfd9ec554bec88eefde1811fc3a5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34197 } }
I20260924 11:52:09.954425 20114 leader_election.cc:302] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3618dfd9ec554bec88eefde1811fc3a5; no voters:
I20260924 11:52:09.954977 20114 leader_election.cc:288] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:09.955132 20119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:09.955809 20119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3618dfd9ec554bec88eefde1811fc3a5, State: Running, Role: LEADER
I20260924 11:52:09.956720 20119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3618dfd9ec554bec88eefde1811fc3a5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34197 } }
I20260924 11:52:09.958514 20114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:09.959800 20121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3618dfd9ec554bec88eefde1811fc3a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3618dfd9ec554bec88eefde1811fc3a5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34197 } } }
I20260924 11:52:09.960181 20121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:09.960572 20120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3618dfd9ec554bec88eefde1811fc3a5. Latest consensus state: current_term: 1 leader_uuid: "3618dfd9ec554bec88eefde1811fc3a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3618dfd9ec554bec88eefde1811fc3a5" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 34197 } } }
I20260924 11:52:09.960945 20120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:09.961112 20123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:09.965826 20123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:09.979746 20123 catalog_manager.cc:1347] Generated new cluster ID: 31d6c58896e3484cb577963b09b52831
I20260924 11:52:09.979876 20123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:10.003017 20123 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:10.004940 20123 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:10.011726 20123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5: Generated new TSK 0
I20260924 11:52:10.012816 20123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:52:10.068470 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34197
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:10.410280 20140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:10.410554 20140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:10.410646 20140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:10.417737 20140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:10.417840 20140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:10.417948 20140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:52:10.427402 20140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:10.428627 20140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:10.429868 20140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:10.442883 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:10.442977 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:10.443008 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:10.444018 20140 server_base.cc:1034] running on GCE node
I20260924 11:52:10.444613 20140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:10.445611 20140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:10.446862 20140 hybrid_clock.cc:648] HybridClock initialized: now 1790250730446815 us; error 39 us; skew 500 ppm
I20260924 11:52:10.447185 20140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:10 dist-test-slave-dd32 krb5kdc[20034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250730, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:10.450330 20140 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:52:10.451838 20140 webserver.cc:460] Webserver started at http://127.1.210.1:36001/ using document root <none> and password file <none>
I20260924 11:52:10.452530 20140 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:10.452680 20140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:10.452991 20140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:10.455839 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "07928d452f964e6986df3f203eaa95ad"
format_stamp: "Formatted at 2026-09-24 11:52:10 on dist-test-slave-dd32"
server_key: "c631316292337de6ae18c7eb0e2b9a55"
server_key_iv: "0fb63cbb317a213d3458ca8146485735"
server_key_version: "encryptionkey@0"
I20260924 11:52:10.456625 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "07928d452f964e6986df3f203eaa95ad"
format_stamp: "Formatted at 2026-09-24 11:52:10 on dist-test-slave-dd32"
server_key: "c631316292337de6ae18c7eb0e2b9a55"
server_key_iv: "0fb63cbb317a213d3458ca8146485735"
server_key_version: "encryptionkey@0"
I20260924 11:52:10.462143 20140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:52:10.465565 20155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:10.466823 20140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:52:10.467051 20140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "07928d452f964e6986df3f203eaa95ad"
format_stamp: "Formatted at 2026-09-24 11:52:10 on dist-test-slave-dd32"
server_key: "c631316292337de6ae18c7eb0e2b9a55"
server_key_iv: "0fb63cbb317a213d3458ca8146485735"
server_key_version: "encryptionkey@0"
I20260924 11:52:10.467276 20140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:10.490762 20140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:10.495342 20140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:10.495651 20140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:10.496690 20140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:10.498095 20140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:10.498204 20140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:10.498353 20140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:10.498425 20140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:10.530663 20140 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:33509
I20260924 11:52:10.530750 20267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:33509 every 8 connection(s)
I20260924 11:52:10.531987 20140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:52:10.532984 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20140
I20260924 11:52:10.533203 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:52:10.533721 1864 external_mini_cluster.cc:1428] Setting key ec1b1b48b81957cc8432edc12401b07f
Sep 24 11:52:10 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250730, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:10.553638 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:10.534692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:36087 (local address 127.1.210.62:34197)
0924 11:52:10.535203 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:10.535208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:10.536390 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 11:52:10.538250 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:10.538257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:10.538349 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:10.538462 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:10.539834 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:10.541051 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:10.542571 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:10.543307 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:10.543421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:10.547750 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:10.547755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:10.547758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:10.547788 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:10.550447 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:10.551357 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:10.551360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:10.551363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:10.551472 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:10.551939 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:10.551942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:10.551944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:10.552183 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:10.552290 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:10.553208 (+ 918us) server_negotiation.cc:299] Negotiation successful
0924 11:52:10.553371 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":279,"threads_started":1}
I20260924 11:52:10.555187 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:10.535314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:34197 (local address 127.1.210.1:36087)
0924 11:52:10.536176 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:10.536226 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:10.537988 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:10.538644 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:10.538651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:10.539134 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:10.539663 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:10.539678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:10.541213 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:10.541217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:10.542421 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:10.542432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:10.543548 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:10.543556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:10.543867 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:10.544604 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:10.544636 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:10.547551 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:10.550584 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:10.550596 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:10.550613 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:10.551247 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:10.551584 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:10.551592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:10.551598 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:10.551832 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:10.552310 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:10.552321 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:10.552765 (+ 444us) client_negotiation.cc:769] Sending connection context
0924 11:52:10.553118 (+ 353us) client_negotiation.cc:240] Negotiation successful
0924 11:52:10.553676 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":384,"threads_started":1}
I20260924 11:52:10.556919 20268 heartbeater.cc:347] Connected to a master server at 127.1.210.62:34197
I20260924 11:52:10.557291 20268 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:10.558000 20268 heartbeater.cc:511] Master 127.1.210.62:34197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
I20260924 11:52:10.560284 20079 ts_manager.cc:194] Registered new tserver with Master: 07928d452f964e6986df3f203eaa95ad (127.1.210.1:33509)
I20260924 11:52:10.561851 20079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:36087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:52:10.592823 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34197
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:10.914384 20274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:10.914726 20274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:10.914938 20274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:10.922135 20274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:10.922297 20274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:10.922482 20274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:52:10.935860 20274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:10.937314 20274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:10.938689 20274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:10.951700 20280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:10.951748 20279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:10.951771 20282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:10.952399 20274 server_base.cc:1034] running on GCE node
I20260924 11:52:10.952971 20274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:10.953953 20274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:10.955197 20274 hybrid_clock.cc:648] HybridClock initialized: now 1790250730955150 us; error 52 us; skew 500 ppm
I20260924 11:52:10.955520 20274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:10 dist-test-slave-dd32 krb5kdc[20034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250730, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:10.958671 20274 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:52:10.960215 20274 webserver.cc:460] Webserver started at http://127.1.210.2:43659/ using document root <none> and password file <none>
I20260924 11:52:10.960902 20274 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:10.961055 20274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:10.961396 20274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:10.964251 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "3e8061e3d2944129ad702fd21caafc00"
format_stamp: "Formatted at 2026-09-24 11:52:10 on dist-test-slave-dd32"
server_key: "8cac40ea38c016160c7b04c761494356"
server_key_iv: "e6e1b9662afdf84b4f6ba40e188b9603"
server_key_version: "encryptionkey@0"
I20260924 11:52:10.965027 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8061e3d2944129ad702fd21caafc00"
format_stamp: "Formatted at 2026-09-24 11:52:10 on dist-test-slave-dd32"
server_key: "8cac40ea38c016160c7b04c761494356"
server_key_iv: "e6e1b9662afdf84b4f6ba40e188b9603"
server_key_version: "encryptionkey@0"
I20260924 11:52:10.970081 20274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:52:10.974004 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:10.975567 20274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:10.975749 20274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "3e8061e3d2944129ad702fd21caafc00"
format_stamp: "Formatted at 2026-09-24 11:52:10 on dist-test-slave-dd32"
server_key: "8cac40ea38c016160c7b04c761494356"
server_key_iv: "e6e1b9662afdf84b4f6ba40e188b9603"
server_key_version: "encryptionkey@0"
I20260924 11:52:10.975948 20274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:11.001930 20274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:11.006265 20274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:11.006584 20274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:11.007540 20274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:11.008860 20274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:11.008970 20274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:11.009171 20274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:11.009269 20274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:11.042168 20274 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:43221
I20260924 11:52:11.042222 20401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:43221 every 8 connection(s)
I20260924 11:52:11.043457 20274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:52:11.045637 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20274
I20260924 11:52:11.045820 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:52:11.046399 1864 external_mini_cluster.cc:1428] Setting key a6866ac012ea3c3c26512eed4b63697c
Sep 24 11:52:11 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250730, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:11.064810 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.046173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:51681 (local address 127.1.210.62:34197)
0924 11:52:11.046391 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.046397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.047638 (+ 1241us) server_negotiation.cc:407] Connection header received
0924 11:52:11.049550 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.049559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.049654 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.049766 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:11.051221 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.052333 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.053927 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.054671 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.054819 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.059003 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:11.059008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:11.059012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:11.059043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:11.061810 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:11.062533 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:11.062539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:11.062547 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:11.062674 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:11.063101 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:11.063106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:11.063116 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:11.063368 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:11.063526 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.064220 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.064449 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:52:11.065776 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.046682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:34197 (local address 127.1.210.2:51681)
0924 11:52:11.047466 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.047512 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.049290 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.049976 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.049984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.050485 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:11.051044 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.051060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.052470 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.052473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.053781 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.053797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.054873 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.054877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.055069 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.055759 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:11.055779 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:11.058749 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:11.061957 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:11.061965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:11.061981 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:11.062401 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:11.062788 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:11.062794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:11.062798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:11.062980 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:11.063507 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:11.063514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:11.063840 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.064100 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.064640 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":344,"threads_started":1}
I20260924 11:52:11.067385 20402 heartbeater.cc:347] Connected to a master server at 127.1.210.62:34197
I20260924 11:52:11.067703 20402 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:11.068391 20402 heartbeater.cc:511] Master 127.1.210.62:34197 requested a full tablet report, sending...
I20260924 11:52:11.069995 20079 ts_manager.cc:194] Registered new tserver with Master: 3e8061e3d2944129ad702fd21caafc00 (127.1.210.2:43221)
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
I20260924 11:52:11.071230 20079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:51681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:52:11.104552 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34197
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:11.429736 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:11.430042 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:11.430249 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:11.437949 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:11.438093 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:11.438284 20407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:52:11.448233 20407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:11.449663 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:11.451023 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:11.464855 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:11.464888 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:11.464787 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:11.465639 20407 server_base.cc:1034] running on GCE node
I20260924 11:52:11.466326 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:11.467221 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:11.468451 20407 hybrid_clock.cc:648] HybridClock initialized: now 1790250731468391 us; error 101 us; skew 500 ppm
I20260924 11:52:11.468762 20407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:11 dist-test-slave-dd32 krb5kdc[20034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250731, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:11.471890 20407 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:52:11.473531 20407 webserver.cc:460] Webserver started at http://127.1.210.3:40073/ using document root <none> and password file <none>
I20260924 11:52:11.474186 20407 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:11.474284 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:11.474603 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:11.477753 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "71a1ab50f28948e68cfa578bb03b0d5a"
format_stamp: "Formatted at 2026-09-24 11:52:11 on dist-test-slave-dd32"
server_key: "cb06c7eaf97ddf98a705dd0d9bb9cc51"
server_key_iv: "b36195c265934f1026bf4fe14d08376d"
server_key_version: "encryptionkey@0"
I20260924 11:52:11.478502 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "71a1ab50f28948e68cfa578bb03b0d5a"
format_stamp: "Formatted at 2026-09-24 11:52:11 on dist-test-slave-dd32"
server_key: "cb06c7eaf97ddf98a705dd0d9bb9cc51"
server_key_iv: "b36195c265934f1026bf4fe14d08376d"
server_key_version: "encryptionkey@0"
I20260924 11:52:11.483784 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:52:11.487124 20422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:11.488385 20407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:52:11.488552 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "71a1ab50f28948e68cfa578bb03b0d5a"
format_stamp: "Formatted at 2026-09-24 11:52:11 on dist-test-slave-dd32"
server_key: "cb06c7eaf97ddf98a705dd0d9bb9cc51"
server_key_iv: "b36195c265934f1026bf4fe14d08376d"
server_key_version: "encryptionkey@0"
I20260924 11:52:11.488752 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:11.509414 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:11.513233 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:11.513510 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:11.514449 20407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:11.515743 20407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:11.515818 20407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:11.515985 20407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:11.516041 20407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:11.552946 20407 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:38557
I20260924 11:52:11.552985 20534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:38557 every 8 connection(s)
I20260924 11:52:11.554297 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:52:11.557551 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20407
I20260924 11:52:11.557755 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:52:11.558308 1864 external_mini_cluster.cc:1428] Setting key e12cedc0d357f5b28d2ff727b193e67b
I20260924 11:52:11.565521 20268 heartbeater.cc:503] Master 127.1.210.62:34197 was elected leader, sending a full tablet report...
Sep 24 11:52:11 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250731, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:11.575207 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.556769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:53471 (local address 127.1.210.62:34197)
0924 11:52:11.556958 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.556962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.558381 (+ 1419us) server_negotiation.cc:407] Connection header received
0924 11:52:11.560612 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.560618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.560732 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.560874 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:11.562464 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.563239 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.564734 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.565174 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.565301 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.569055 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:11.569110 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:11.569115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:11.569156 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:11.572014 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:11.572894 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:11.572899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:11.572902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:11.573093 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:11.573630 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:11.573635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:11.573637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:11.573965 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:11.574070 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.574797 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.574950 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:52:11.576160 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.557336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:34197 (local address 127.1.210.3:53471)
0924 11:52:11.558184 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.558231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.560355 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.561208 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.561220 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.561743 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:11.562313 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.562330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.563378 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.563382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.564610 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.564623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.565384 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.565388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.565581 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.566401 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:11.566435 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:11.568852 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:11.572168 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:11.572176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:11.572192 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:11.572762 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:11.573228 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:11.573232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:11.573235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:11.573501 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:11.574105 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:11.574111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:11.574418 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.574715 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.575078 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":343,"threads_started":1}
I20260924 11:52:11.577406 20535 heartbeater.cc:347] Connected to a master server at 127.1.210.62:34197
I20260924 11:52:11.577700 20535 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:11.578382 20535 heartbeater.cc:511] Master 127.1.210.62:34197 requested a full tablet report, sending...
I20260924 11:52:11.579742 20079 ts_manager.cc:194] Registered new tserver with Master: 71a1ab50f28948e68cfa578bb03b0d5a (127.1.210.3:38557)
I20260924 11:52:11.580478 20079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:53471
I20260924 11:52:11.585759 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:52:11.602021 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.589659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.1.210.62:34197)
0924 11:52:11.589886 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.589896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.590187 (+ 291us) server_negotiation.cc:407] Connection header received
0924 11:52:11.590307 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.590311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.590366 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.590444 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:11.591772 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.592577 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.594032 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.594422 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.594519 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.596941 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:11.596946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:11.596949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:11.596976 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:11.599511 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:11.600192 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:11.600195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:11.600197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:11.600340 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:11.600692 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:11.600695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:11.600697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:11.600965 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:11.601254 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.601533 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.601702 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:52:11.605796 20079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:52:11.612537 20079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:52:11.649294 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.625030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.1.210.2:43221)
0924 11:52:11.638675 (+ 13645us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.638686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.639957 (+ 1271us) server_negotiation.cc:407] Connection header received
0924 11:52:11.640235 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.640245 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.640508 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.640684 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:11.642562 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.643916 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.646954 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.648146 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.648313 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.648513 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.648781 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.648948 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13392,"thread_start_us":13634,"threads_started":1}
I20260924 11:52:11.649410 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.624281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:43221 (local address 127.0.0.1:43144)
0924 11:52:11.639846 (+ 15565us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.639863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.640055 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.641030 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.641038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.641119 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:11.642372 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.642387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.644055 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.644062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.646796 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.646810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.648344 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.648350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.648530 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.648590 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.648724 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.648912 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15448,"mutex_wait_us":845,"spinlock_wait_cycles":78336,"thread_start_us":15331,"threads_started":1}
I20260924 11:52:11.649996 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.626132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:39698)
0924 11:52:11.641638 (+ 15506us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.641657 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.641764 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.642950 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.642956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.642987 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:11.643317 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.643328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.644952 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.644958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.647138 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.647153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.649183 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.649188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.649301 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.649323 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.649442 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.649585 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15322,"mutex_wait_us":24,"thread_start_us":15174,"threads_started":1}
I20260924 11:52:11.650521 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.624147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38557 (local address 127.0.0.1:49946)
0924 11:52:11.639193 (+ 15046us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.639244 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.639438 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.641030 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.641038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.641119 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:11.641697 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.641713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.644326 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.644337 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.646977 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.647004 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.649667 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.649674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.649868 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.649901 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.649989 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.650113 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14750,"mutex_wait_us":400,"spinlock_wait_cycles":21632,"thread_start_us":15490,"threads_started":1}
I20260924 11:52:11.650661 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.626790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.1.210.1:33509)
0924 11:52:11.642351 (+ 15561us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.642361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.642385 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:52:11.642485 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.642495 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.642741 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.642949 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:11.643505 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.644804 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.647323 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.648649 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.648758 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.648977 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.650220 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.650369 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15362,"thread_start_us":15189,"threads_started":1}
I20260924 11:52:11.651185 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.624335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.1.210.3:38557)
0924 11:52:11.639396 (+ 15061us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.639404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.639438 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:52:11.640350 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.640356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.640526 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.640684 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:11.642579 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.644161 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.647142 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.648592 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.648763 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.648981 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.650829 (+ 1848us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.650958 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14885,"thread_start_us":14763,"threads_started":1}
I20260924 11:52:11.653862 20337 tablet_service.cc:1467] Processing CreateTablet for tablet bdb3dd25fd1f49179da679fa5fcdb5fb (DEFAULT_TABLE table=test-table [id=ff324dccda454b13a91d18783c82cd87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:11.654161 20203 tablet_service.cc:1467] Processing CreateTablet for tablet bdb3dd25fd1f49179da679fa5fcdb5fb (DEFAULT_TABLE table=test-table [id=ff324dccda454b13a91d18783c82cd87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:11.654179 20470 tablet_service.cc:1467] Processing CreateTablet for tablet bdb3dd25fd1f49179da679fa5fcdb5fb (DEFAULT_TABLE table=test-table [id=ff324dccda454b13a91d18783c82cd87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:11.655614 20203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb3dd25fd1f49179da679fa5fcdb5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:11.655606 20470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb3dd25fd1f49179da679fa5fcdb5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:11.656306 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb3dd25fd1f49179da679fa5fcdb5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:11.667275 20553 tablet_bootstrap.cc:492] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Bootstrap starting.
I20260924 11:52:11.671157 20555 tablet_bootstrap.cc:492] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Bootstrap starting.
I20260924 11:52:11.672194 20553 tablet_bootstrap.cc:654] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:11.672281 20554 tablet_bootstrap.cc:492] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Bootstrap starting.
I20260924 11:52:11.673621 20553 log.cc:824] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:11.675745 20554 tablet_bootstrap.cc:654] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:11.676100 20555 tablet_bootstrap.cc:654] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:11.677080 20553 tablet_bootstrap.cc:492] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: No bootstrap required, opened a new log
I20260924 11:52:11.676999 20554 log.cc:824] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:11.677228 20555 log.cc:824] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:11.677353 20553 ts_tablet_manager.cc:1397] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 11:52:11.683138 20554 tablet_bootstrap.cc:492] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: No bootstrap required, opened a new log
I20260924 11:52:11.683321 20555 tablet_bootstrap.cc:492] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: No bootstrap required, opened a new log
I20260924 11:52:11.683465 20554 ts_tablet_manager.cc:1397] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 11:52:11.683594 20555 ts_tablet_manager.cc:1397] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 11:52:11.688828 20553 raft_consensus.cc:348] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.689313 20553 raft_consensus.cc:374] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:11.689424 20553 raft_consensus.cc:729] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a1ab50f28948e68cfa578bb03b0d5a, State: Initialized, Role: FOLLOWER
I20260924 11:52:11.690058 20553 consensus_queue.cc:260] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.691207 20535 heartbeater.cc:503] Master 127.1.210.62:34197 was elected leader, sending a full tablet report...
I20260924 11:52:11.691839 20553 ts_tablet_manager.cc:1428] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 11:52:11.692102 20554 raft_consensus.cc:348] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.692507 20554 raft_consensus.cc:374] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:11.692611 20554 raft_consensus.cc:729] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8061e3d2944129ad702fd21caafc00, State: Initialized, Role: FOLLOWER
I20260924 11:52:11.693254 20554 consensus_queue.cc:260] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.694010 20555 raft_consensus.cc:348] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.694419 20555 raft_consensus.cc:374] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:11.694537 20555 raft_consensus.cc:729] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07928d452f964e6986df3f203eaa95ad, State: Initialized, Role: FOLLOWER
I20260924 11:52:11.694917 20554 ts_tablet_manager.cc:1428] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260924 11:52:11.695111 20555 consensus_queue.cc:260] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.695433 20402 heartbeater.cc:503] Master 127.1.210.62:34197 was elected leader, sending a full tablet report...
I20260924 11:52:11.699240 20555 ts_tablet_manager.cc:1428] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260924 11:52:11.784317 20269 tablet.cc:2369] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:52:11.795465 20403 tablet.cc:2369] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:52:11.805918 20536 tablet.cc:2369] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:11.894989 20560 raft_consensus.cc:482] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:52:11.895330 20560 raft_consensus.cc:504] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.897019 20560 leader_election.cc:288] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a1ab50f28948e68cfa578bb03b0d5a (127.1.210.3:38557), 07928d452f964e6986df3f203eaa95ad (127.1.210.1:33509)
I20260924 11:52:11.905537 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.897594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:45401 (local address 127.1.210.3:38557)
0924 11:52:11.898067 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.898074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.898254 (+ 180us) server_negotiation.cc:407] Connection header received
0924 11:52:11.898440 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.898444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.898505 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.898614 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:11.899187 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.900283 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.903108 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.904380 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.904505 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.904609 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.905164 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.905287 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260924 11:52:11.905537 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.897445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:50771 (local address 127.1.210.1:33509)
0924 11:52:11.897874 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.897884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.898507 (+ 623us) server_negotiation.cc:407] Connection header received
0924 11:52:11.898697 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.898701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.898778 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.898856 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:11.899474 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.900502 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.903109 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.904364 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.904505 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.904609 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.905153 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.905282 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260924 11:52:11.905766 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.897274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38557 (local address 127.1.210.2:45401)
0924 11:52:11.898100 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.898138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.898301 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.898604 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.898611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.898636 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:11.899031 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.899049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.900420 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.900423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.902939 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.902953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.904654 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.904660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.904948 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.905001 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.905134 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.905286 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":225,"thread_start_us":459,"threads_started":1}
I20260924 11:52:11.905789 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.897462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.1.210.2:50771)
0924 11:52:11.898397 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:11.898414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:11.898521 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:11.898904 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:11.898909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:11.898939 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:11.899321 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.899329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.900640 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.900646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.902938 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:11.902953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.904668 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.904673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:11.904944 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.904999 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 11:52:11.905128 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 11:52:11.905286 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":89,"thread_start_us":571,"threads_started":1}
I20260924 11:52:11.907039 20490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb3dd25fd1f49179da679fa5fcdb5fb" candidate_uuid: "3e8061e3d2944129ad702fd21caafc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" is_pre_election: true
I20260924 11:52:11.907039 20223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb3dd25fd1f49179da679fa5fcdb5fb" candidate_uuid: "3e8061e3d2944129ad702fd21caafc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07928d452f964e6986df3f203eaa95ad" is_pre_election: true
I20260924 11:52:11.907636 20490 raft_consensus.cc:2454] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e8061e3d2944129ad702fd21caafc00 in term 0.
I20260924 11:52:11.907624 20223 raft_consensus.cc:2454] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e8061e3d2944129ad702fd21caafc00 in term 0.
I20260924 11:52:11.908309 20290 leader_election.cc:302] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8061e3d2944129ad702fd21caafc00, 71a1ab50f28948e68cfa578bb03b0d5a; no voters:
I20260924 11:52:11.908664 20560 raft_consensus.cc:2781] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:52:11.908812 20560 raft_consensus.cc:482] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:52:11.908926 20560 raft_consensus.cc:3037] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:11.911109 20560 raft_consensus.cc:504] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.911711 20560 leader_election.cc:288] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [CANDIDATE]: Term 1 election: Requested vote from peers 71a1ab50f28948e68cfa578bb03b0d5a (127.1.210.3:38557), 07928d452f964e6986df3f203eaa95ad (127.1.210.1:33509)
I20260924 11:52:11.912151 20490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb3dd25fd1f49179da679fa5fcdb5fb" candidate_uuid: "3e8061e3d2944129ad702fd21caafc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a1ab50f28948e68cfa578bb03b0d5a"
I20260924 11:52:11.912227 20223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb3dd25fd1f49179da679fa5fcdb5fb" candidate_uuid: "3e8061e3d2944129ad702fd21caafc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07928d452f964e6986df3f203eaa95ad"
I20260924 11:52:11.912341 20490 raft_consensus.cc:3037] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:11.912474 20223 raft_consensus.cc:3037] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:11.914613 20490 raft_consensus.cc:2454] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8061e3d2944129ad702fd21caafc00 in term 1.
I20260924 11:52:11.914742 20223 raft_consensus.cc:2454] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8061e3d2944129ad702fd21caafc00 in term 1.
I20260924 11:52:11.915030 20290 leader_election.cc:302] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8061e3d2944129ad702fd21caafc00, 71a1ab50f28948e68cfa578bb03b0d5a; no voters:
I20260924 11:52:11.915308 20560 raft_consensus.cc:2781] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:11.915766 20560 raft_consensus.cc:686] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 1 LEADER]: Becoming Leader. State: Replica: 3e8061e3d2944129ad702fd21caafc00, State: Running, Role: LEADER
I20260924 11:52:11.916345 20560 consensus_queue.cc:237] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } }
I20260924 11:52:11.920936 20079 catalog_manager.cc:5495] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e8061e3d2944129ad702fd21caafc00 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "3e8061e3d2944129ad702fd21caafc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8061e3d2944129ad702fd21caafc00" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 43221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 } health_report { overall_health: UNKNOWN } } }
I20260924 11:52:11.943270 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.938419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.1.210.2:43221)
0924 11:52:11.938597 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.938602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.938629 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:52:11.938798 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.938802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.938873 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.938991 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:52:11.939328 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.940020 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.941397 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:11.941784 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:11.941873 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:11.942313 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:52:11.942779 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:52:11.942912 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:11.943041 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 11:52:11.943078 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:52:11.951220 20490 raft_consensus.cc:1261] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 1 FOLLOWER]: Refusing update from remote peer 3e8061e3d2944129ad702fd21caafc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:11.951220 20223 raft_consensus.cc:1261] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 1 FOLLOWER]: Refusing update from remote peer 3e8061e3d2944129ad702fd21caafc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:11.952313 20564 consensus_queue.cc:1035] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a1ab50f28948e68cfa578bb03b0d5a" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:11.952694 20560 consensus_queue.cc:1035] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07928d452f964e6986df3f203eaa95ad" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:11.967391 20569 mvcc.cc:205] Tried to move back new op lower bound from 7332866998061314048 to 7332866997933875200. Current Snapshot: MvccSnapshot[applied={T|T < 7332866998061314048}]
I20260924 11:52:11.971520 20568 mvcc.cc:205] Tried to move back new op lower bound from 7332866998061314048 to 7332866997933875200. Current Snapshot: MvccSnapshot[applied={T|T < 7332866998061314048}]
I20260924 11:52:11.980571 20079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:11.980850 20079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:11.984532 20079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3618dfd9ec554bec88eefde1811fc3a5: Sending DeleteTablet for 3 replicas of tablet bdb3dd25fd1f49179da679fa5fcdb5fb
I20260924 11:52:11.985630 20470 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb3dd25fd1f49179da679fa5fcdb5fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:11 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260924 11:52:11.986086 20203 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb3dd25fd1f49179da679fa5fcdb5fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:11 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:39698
I20260924 11:52:11.986347 20337 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb3dd25fd1f49179da679fa5fcdb5fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:11 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:43144
I20260924 11:52:11.990327 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20140
I20260924 11:52:12.002126 20578 tablet_replica.cc:330] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: stopping tablet replica
I20260924 11:52:12.002768 20578 raft_consensus.cc:2229] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:12.003458 20578 raft_consensus.cc:2258] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:12.004561 20579 tablet_replica.cc:330] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: stopping tablet replica
I20260924 11:52:12.005302 20579 raft_consensus.cc:2229] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:52:12.006932 20579 raft_consensus.cc:2258] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:12.007748 20577 tablet_replica.cc:330] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: stopping tablet replica
I20260924 11:52:12.008253 20577 raft_consensus.cc:2229] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:12.008797 20577 raft_consensus.cc:2258] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:12.011092 20579 ts_tablet_manager.cc:1905] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:12.011178 20577 ts_tablet_manager.cc:1905] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:52:12 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:52:12.018914 20579 ts_tablet_manager.cc:1918] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:12.019106 20579 log.cc:1196] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/bdb3dd25fd1f49179da679fa5fcdb5fb
I20260924 11:52:12.019083 20577 ts_tablet_manager.cc:1918] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:12.019273 20577 log.cc:1196] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/bdb3dd25fd1f49179da679fa5fcdb5fb
I20260924 11:52:12.019718 20579 ts_tablet_manager.cc:1939] T bdb3dd25fd1f49179da679fa5fcdb5fb P 3e8061e3d2944129ad702fd21caafc00: Deleting consensus metadata
I20260924 11:52:12.019773 20577 ts_tablet_manager.cc:1939] T bdb3dd25fd1f49179da679fa5fcdb5fb P 71a1ab50f28948e68cfa578bb03b0d5a: Deleting consensus metadata
I20260924 11:52:12.021839 20063 catalog_manager.cc:4841] TS 71a1ab50f28948e68cfa578bb03b0d5a (127.1.210.3:38557): tablet bdb3dd25fd1f49179da679fa5fcdb5fb (table test-table [id=ff324dccda454b13a91d18783c82cd87]) successfully deleted
I20260924 11:52:12.021931 20578 ts_tablet_manager.cc:1905] T bdb3dd25fd1f49179da679fa5fcdb5fb P 07928d452f964e6986df3f203eaa95ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:12.022790 20065 catalog_manager.cc:4841] TS 3e8061e3d2944129ad702fd21caafc00 (127.1.210.2:43221): tablet bdb3dd25fd1f49179da679fa5fcdb5fb (table test-table [id=ff324dccda454b13a91d18783c82cd87]) successfully deleted
I20260924 11:52:12.023790 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:11.991642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.1.210.1:33509)
0924 11:52:11.991819 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:11.991825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:11.997237 (+ 5412us) server_negotiation.cc:407] Connection header received
0924 11:52:11.997502 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:11.997519 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:11.997641 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:11.997760 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:12.004096 (+ 6336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:12.005367 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:12.007597 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:12.008305 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:12.008420 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:12.014962 (+ 6542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:12.014982 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:12.015000 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:12.015056 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:12.018949 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:12.020331 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:12.020341 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:12.020346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:12.020549 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:12.021112 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:12.021119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:12.021123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:12.021570 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:12.021890 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:12.022811 (+ 921us) server_negotiation.cc:299] Negotiation successful
0924 11:52:12.023076 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:52:12.024570 20263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:12.130511 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20140
W20260924 11:52:12.153219 20064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.1:33509 (error 108)
I20260924 11:52:12.153333 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20274
I20260924 11:52:12.155645 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.154822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:39714)
0924 11:52:12.155384 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.155485 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":350,"threads_started":1}
W20260924 11:52:12.155943 20064 catalog_manager.cc:4584] TS 07928d452f964e6986df3f203eaa95ad (127.1.210.1:33509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdb3dd25fd1f49179da679fa5fcdb5fb: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Sep 24 11:52:12 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:52:12.171176 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.154380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.1.210.2:43221)
0924 11:52:12.154583 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:12.154593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:12.154613 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:52:12.154708 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:12.154713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:12.154792 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:12.154896 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:12.156680 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:12.157642 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:12.159583 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:12.160005 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:12.160106 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:12.165207 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:12.165219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:12.165234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:12.165265 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:12.168418 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:12.169101 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:12.169113 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:12.169117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:12.169261 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:12.169639 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:12.169649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:12.169652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:12.169991 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:12.170317 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:12.170599 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 11:52:12.170841 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:52:12.171962 20397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:12.209753 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.209210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:39718)
0924 11:52:12.209444 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.209575 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 11:52:12.210827 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.210419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:39734)
0924 11:52:12.210571 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.210651 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:52:12.271322 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.270714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:39748)
0924 11:52:12.270987 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.271118 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 11:52:12.272539 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.272092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:39758)
0924 11:52:12.272273 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.272363 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:52:12.288424 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20274
I20260924 11:52:12.307554 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20407
Sep 24 11:52:12 dist-test-slave-dd32 krb5kdc[20034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:52:12.322237 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.308303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.1.210.3:38557)
0924 11:52:12.308490 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:12.308497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:12.308516 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:52:12.308741 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:12.308751 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:12.308816 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:12.308915 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:12.310265 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:12.310977 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:12.312281 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:12.312672 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:12.312804 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:12.317267 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:12.317281 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:12.317298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:12.317335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:12.319865 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:12.320470 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:12.320476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:12.320478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:12.320621 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:12.320915 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:12.320917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:12.320919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:12.321336 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:12.321544 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:12.321783 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 11:52:12.321944 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:52:12.322898 20530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:12.353250 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.352642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:40510)
0924 11:52:12.352890 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.353004 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 11:52:12.354437 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.353929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:40524)
0924 11:52:12.354178 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.354274 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 11:52:12.413800 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20407
I20260924 11:52:12.430181 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20047
I20260924 11:52:12.431432 20109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:12.544256 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:12.543355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:33509 (local address 127.0.0.1:40538)
0924 11:52:12.543957 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:12.544076 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.1:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260924 11:52:12.544327 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20047
2026-09-24T11:52:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4434 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T11:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:12Z Disabled control of system clock
I20260924 11:52:12.614522 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:46065
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:46065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:52:12.946137 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:12.946426 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:12.946537 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:12.954016 20590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:12.954135 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:12.954178 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:12.954243 20590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:12.954295 20590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:12.963819 20590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:46065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:46065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:12.965158 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:12.966933 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:12.978788 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:12.978858 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:12.979023 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:12.980019 20590 server_base.cc:1034] running on GCE node
I20260924 11:52:12.980825 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:12.982306 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:12.983546 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790250732983489 us; error 45 us; skew 500 ppm
I20260924 11:52:12.983901 20590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:12.986171 20590 webserver.cc:460] Webserver started at http://127.1.210.62:37705/ using document root <none> and password file <none>
I20260924 11:52:12.986845 20590 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:12.986987 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:12.987324 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:12.990214 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "bffde95f162746829a6215a407d08b62"
format_stamp: "Formatted at 2026-09-24 11:52:12 on dist-test-slave-dd32"
server_key: "d71bf3bc34a08e3a9cf88a5c390be8db"
server_key_iv: "81195ad63d07ce63ed8699e370b5932a"
server_key_version: "encryptionkey@0"
I20260924 11:52:12.991056 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "bffde95f162746829a6215a407d08b62"
format_stamp: "Formatted at 2026-09-24 11:52:12 on dist-test-slave-dd32"
server_key: "d71bf3bc34a08e3a9cf88a5c390be8db"
server_key_iv: "81195ad63d07ce63ed8699e370b5932a"
server_key_version: "encryptionkey@0"
I20260924 11:52:12.995967 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:52:12.999368 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:13.000607 20590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:52:13.000804 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "bffde95f162746829a6215a407d08b62"
format_stamp: "Formatted at 2026-09-24 11:52:12 on dist-test-slave-dd32"
server_key: "d71bf3bc34a08e3a9cf88a5c390be8db"
server_key_iv: "81195ad63d07ce63ed8699e370b5932a"
server_key_version: "encryptionkey@0"
I20260924 11:52:13.001025 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:13.021880 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:13.022730 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:13.023036 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:13.041868 20590 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:46065
I20260924 11:52:13.041877 20655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:46065 every 8 connection(s)
I20260924 11:52:13.043252 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:13.046398 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20590
I20260924 11:52:13.046569 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:13.047030 1864 external_mini_cluster.cc:1428] Setting key fd31d9961e8aa410b6d2a0761321c2f1
I20260924 11:52:13.047714 20656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:13.063912 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:13.049001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.1.210.62:46065)
0924 11:52:13.050182 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:13.050200 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:13.050258 (+ 58us) server_negotiation.cc:407] Connection header received
0924 11:52:13.052816 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:13.052847 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:13.053512 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:13.054015 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:13.054604 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:13.056669 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:13.058301 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:13.059430 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:13.059656 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:13.060859 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:13.060877 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:13.060901 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:13.061741 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:13.061850 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:13.061884 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:13.062016 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:13.062250 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 11:52:13.062588 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":338,"threads_started":1}
I20260924 11:52:13.064105 20656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62: Bootstrap starting.
I20260924 11:52:13.068082 20656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:13.069056 20656 log.cc:824] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:13.072292 20656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62: No bootstrap required, opened a new log
I20260924 11:52:13.080406 20656 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffde95f162746829a6215a407d08b62" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46065 } }
I20260924 11:52:13.080808 20656 raft_consensus.cc:374] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:13.080890 20656 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffde95f162746829a6215a407d08b62, State: Initialized, Role: FOLLOWER
I20260924 11:52:13.081559 20656 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffde95f162746829a6215a407d08b62" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46065 } }
I20260924 11:52:13.081774 20656 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:13.081887 20656 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:13.082028 20656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:13.084736 20656 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffde95f162746829a6215a407d08b62" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46065 } }
I20260924 11:52:13.085323 20656 leader_election.cc:302] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffde95f162746829a6215a407d08b62; no voters:
I20260924 11:52:13.085870 20656 leader_election.cc:288] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:13.086054 20661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:13.086820 20661 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [term 1 LEADER]: Becoming Leader. State: Replica: bffde95f162746829a6215a407d08b62, State: Running, Role: LEADER
I20260924 11:52:13.087347 20661 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffde95f162746829a6215a407d08b62" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46065 } }
I20260924 11:52:13.088140 20656 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:13.090409 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffde95f162746829a6215a407d08b62. Latest consensus state: current_term: 1 leader_uuid: "bffde95f162746829a6215a407d08b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffde95f162746829a6215a407d08b62" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46065 } } }
I20260924 11:52:13.090799 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:13.093150 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bffde95f162746829a6215a407d08b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffde95f162746829a6215a407d08b62" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46065 } } }
I20260924 11:52:13.093979 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:13.094971 20667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:13.100332 20667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:13.109760 20667 catalog_manager.cc:1347] Generated new cluster ID: e5198a5672e7417ca5aa24ed35251772
I20260924 11:52:13.109898 20667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:13.127588 20667 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:13.129050 20667 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:13.139930 20667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62: Generated new TSK 0
I20260924 11:52:13.140853 20667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:13.149983 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:46065
--builtin_ntp_servers=127.1.210.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:52:13.518432 20680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:13.518726 20680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:13.518819 20680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:13.526250 20680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:13.526407 20680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:13.526599 20680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:52:13.535746 20680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:46065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:13.537092 20680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:13.538944 20680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:13.553048 20685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:13.553133 20688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:13.552969 20686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:13.554229 20680 server_base.cc:1034] running on GCE node
I20260924 11:52:13.554899 20680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:13.555825 20680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:13.557149 20680 hybrid_clock.cc:648] HybridClock initialized: now 1790250733557030 us; error 57 us; skew 500 ppm
I20260924 11:52:13.557472 20680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:13.559661 20680 webserver.cc:460] Webserver started at http://127.1.210.1:35121/ using document root <none> and password file <none>
I20260924 11:52:13.560355 20680 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:13.560504 20680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:13.560843 20680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:13.563788 20680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "049dfd097b764214b97f3e05c9aa4296"
format_stamp: "Formatted at 2026-09-24 11:52:13 on dist-test-slave-dd32"
server_key: "e419422a5fe19ef9dbd2b4ec59c6b67a"
server_key_iv: "160e6ccf061d10d7c48178f913e0f9aa"
server_key_version: "encryptionkey@0"
I20260924 11:52:13.564605 20680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "049dfd097b764214b97f3e05c9aa4296"
format_stamp: "Formatted at 2026-09-24 11:52:13 on dist-test-slave-dd32"
server_key: "e419422a5fe19ef9dbd2b4ec59c6b67a"
server_key_iv: "160e6ccf061d10d7c48178f913e0f9aa"
server_key_version: "encryptionkey@0"
I20260924 11:52:13.569670 20680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:52:13.573364 20694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:13.574819 20680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:13.575011 20680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "049dfd097b764214b97f3e05c9aa4296"
format_stamp: "Formatted at 2026-09-24 11:52:13 on dist-test-slave-dd32"
server_key: "e419422a5fe19ef9dbd2b4ec59c6b67a"
server_key_iv: "160e6ccf061d10d7c48178f913e0f9aa"
server_key_version: "encryptionkey@0"
I20260924 11:52:13.575243 20680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:13.605806 20680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:13.606763 20680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:13.607096 20680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:13.608235 20680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:13.609974 20680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:13.610116 20680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:13.610265 20680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:13.610363 20680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:13.646862 20680 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:42441
I20260924 11:52:13.646917 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:42441 every 8 connection(s)
I20260924 11:52:13.648264 20680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:52:13.648941 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20680
I20260924 11:52:13.649137 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:52:13.649695 1864 external_mini_cluster.cc:1428] Setting key ce33680075cbb4d3f1f89ec673ec9c50
I20260924 11:52:13.652796 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:46065
--builtin_ntp_servers=127.1.210.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:52:13.663204 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:13.651216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:48819 (local address 127.1.210.62:46065)
0924 11:52:13.651717 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:13.651723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:13.653367 (+ 1644us) server_negotiation.cc:407] Connection header received
0924 11:52:13.655332 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:13.655340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:13.655434 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:13.655554 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:13.656708 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:13.657909 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:13.659623 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:13.660107 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:13.660219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:13.662037 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:13.662047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:13.662055 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:13.662082 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:13.662145 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:13.662165 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:13.662259 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:13.662629 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 11:52:13.662748 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":250,"threads_started":1}
I20260924 11:52:13.664388 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:13.651997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:46065 (local address 127.1.210.1:48819)
0924 11:52:13.653180 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:13.653237 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:13.655029 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:13.655765 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:13.655774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:13.655866 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:13.656519 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:13.656541 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:13.658086 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:13.658090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:13.659458 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:13.659471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:13.660391 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:13.660399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:13.660778 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:13.661686 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:13.661689 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:13.661742 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:13.661825 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:13.662335 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:13.662347 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:13.662355 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:52:13.662592 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 11:52:13.662973 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":367,"threads_started":1}
I20260924 11:52:13.666316 20807 heartbeater.cc:347] Connected to a master server at 127.1.210.62:46065
I20260924 11:52:13.666667 20807 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:13.667620 20807 heartbeater.cc:511] Master 127.1.210.62:46065 requested a full tablet report, sending...
I20260924 11:52:13.670123 20621 ts_manager.cc:194] Registered new tserver with Master: 049dfd097b764214b97f3e05c9aa4296 (127.1.210.1:42441)
I20260924 11:52:13.671828 20621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.1:48819
W20260924 11:52:13.983412 20811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:13.983701 20811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:13.983790 20811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:13.991317 20811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:13.991417 20811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:13.991528 20811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:52:14.001020 20811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:46065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:14.002282 20811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:14.004050 20811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:14.017138 20817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:14.017194 20816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:14.017203 20819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:14.018195 20811 server_base.cc:1034] running on GCE node
I20260924 11:52:14.018831 20811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:14.019774 20811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:14.020977 20811 hybrid_clock.cc:648] HybridClock initialized: now 1790250734020937 us; error 28 us; skew 500 ppm
I20260924 11:52:14.021332 20811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:14.023629 20811 webserver.cc:460] Webserver started at http://127.1.210.2:34971/ using document root <none> and password file <none>
I20260924 11:52:14.024298 20811 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:14.024389 20811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:14.024668 20811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:14.027469 20811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "133d2bcbdeaf4eb7866410180a3ebf41"
format_stamp: "Formatted at 2026-09-24 11:52:14 on dist-test-slave-dd32"
server_key: "d40008ba726a5e9f5c9e964172394de7"
server_key_iv: "2f27f1f793d2b708f098374bf9a5a996"
server_key_version: "encryptionkey@0"
I20260924 11:52:14.028183 20811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "133d2bcbdeaf4eb7866410180a3ebf41"
format_stamp: "Formatted at 2026-09-24 11:52:14 on dist-test-slave-dd32"
server_key: "d40008ba726a5e9f5c9e964172394de7"
server_key_iv: "2f27f1f793d2b708f098374bf9a5a996"
server_key_version: "encryptionkey@0"
I20260924 11:52:14.033308 20811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:52:14.036710 20825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:14.038100 20811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:14.038230 20811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "133d2bcbdeaf4eb7866410180a3ebf41"
format_stamp: "Formatted at 2026-09-24 11:52:14 on dist-test-slave-dd32"
server_key: "d40008ba726a5e9f5c9e964172394de7"
server_key_iv: "2f27f1f793d2b708f098374bf9a5a996"
server_key_version: "encryptionkey@0"
I20260924 11:52:14.038408 20811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:14.060277 20811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:14.061275 20811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:14.061556 20811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:14.062668 20811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:14.064493 20811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:14.064591 20811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:14.064661 20811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:14.064714 20811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:14.098060 20811 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:44031
I20260924 11:52:14.098141 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:44031 every 8 connection(s)
I20260924 11:52:14.099393 20811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:52:14.099889 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20811
I20260924 11:52:14.100065 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:52:14.100492 1864 external_mini_cluster.cc:1428] Setting key fe2a2290584074b576b4bc6b581367cd
I20260924 11:52:14.103588 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:46065
--builtin_ntp_servers=127.1.210.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:52:14.112869 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.102504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:60361 (local address 127.1.210.62:46065)
0924 11:52:14.102660 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.102664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.104452 (+ 1788us) server_negotiation.cc:407] Connection header received
0924 11:52:14.106426 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.106431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.106522 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.106637 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:14.107709 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.108449 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.110095 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.110483 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.110590 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.112162 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:14.112166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:14.112170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:14.112187 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:14.112219 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:14.112231 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:14.112304 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.112558 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.112635 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 11:52:14.114343 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.103183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:46065 (local address 127.1.210.2:60361)
0924 11:52:14.104301 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.104343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.106152 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.106874 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.106883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.106973 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:14.107534 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.107550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.108606 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.108611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.109907 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.109919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.110720 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.110727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.111047 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.111812 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:14.111818 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:14.111870 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:14.111968 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:14.112331 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:14.112344 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:14.112355 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.112532 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.112947 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":309,"threads_started":1}
I20260924 11:52:14.115754 20938 heartbeater.cc:347] Connected to a master server at 127.1.210.62:46065
I20260924 11:52:14.116083 20938 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:14.116825 20938 heartbeater.cc:511] Master 127.1.210.62:46065 requested a full tablet report, sending...
I20260924 11:52:14.118342 20621 ts_manager.cc:194] Registered new tserver with Master: 133d2bcbdeaf4eb7866410180a3ebf41 (127.1.210.2:44031)
I20260924 11:52:14.119185 20621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.2:60361
W20260924 11:52:14.430611 20941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:14.430949 20941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:14.431142 20941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:14.438367 20941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:14.438534 20941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:14.438755 20941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:52:14.448206 20941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:46065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:14.449555 20941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:14.451306 20941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:14.464550 20946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:14.464641 20947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:14.464711 20949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:14.465870 20941 server_base.cc:1034] running on GCE node
I20260924 11:52:14.466456 20941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:14.467351 20941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:14.468616 20941 hybrid_clock.cc:648] HybridClock initialized: now 1790250734468596 us; error 56 us; skew 500 ppm
I20260924 11:52:14.468935 20941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:14.471220 20941 webserver.cc:460] Webserver started at http://127.1.210.3:46749/ using document root <none> and password file <none>
I20260924 11:52:14.471937 20941 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:14.472092 20941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:14.472426 20941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:14.475322 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "5593f0ab771949d297f3485ecca72bd9"
format_stamp: "Formatted at 2026-09-24 11:52:14 on dist-test-slave-dd32"
server_key: "88064975c6d69d877be8f4eb72ca23a9"
server_key_iv: "9b7fa551650237e342cd214d07bb1772"
server_key_version: "encryptionkey@0"
I20260924 11:52:14.476148 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "5593f0ab771949d297f3485ecca72bd9"
format_stamp: "Formatted at 2026-09-24 11:52:14 on dist-test-slave-dd32"
server_key: "88064975c6d69d877be8f4eb72ca23a9"
server_key_iv: "9b7fa551650237e342cd214d07bb1772"
server_key_version: "encryptionkey@0"
I20260924 11:52:14.481415 20941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:52:14.485162 20955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:14.486524 20941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:14.486722 20941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "5593f0ab771949d297f3485ecca72bd9"
format_stamp: "Formatted at 2026-09-24 11:52:14 on dist-test-slave-dd32"
server_key: "88064975c6d69d877be8f4eb72ca23a9"
server_key_iv: "9b7fa551650237e342cd214d07bb1772"
server_key_version: "encryptionkey@0"
I20260924 11:52:14.486953 20941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:14.505501 20941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:14.506364 20941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:14.506665 20941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:14.507666 20941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:14.509163 20941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:14.509287 20941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:14.509418 20941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:14.509516 20941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:14.544118 20941 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:38277
I20260924 11:52:14.544225 21067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:38277 every 8 connection(s)
I20260924 11:52:14.545583 20941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:52:14.549180 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 20941
I20260924 11:52:14.549362 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:52:14.549821 1864 external_mini_cluster.cc:1428] Setting key a22c635fecfcb7ad51c2dec158e00983
I20260924 11:52:14.558439 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.548129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:39115 (local address 127.1.210.62:46065)
0924 11:52:14.548330 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.548335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.549907 (+ 1572us) server_negotiation.cc:407] Connection header received
0924 11:52:14.552019 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.552025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.552104 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.552215 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:14.553454 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.554192 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.555702 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.556147 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.556288 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.557615 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:14.557620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:14.557623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:14.557640 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:14.557678 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:14.557690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:14.557789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.558019 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.558112 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:52:14.559355 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.548688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:46065 (local address 127.1.210.3:39115)
0924 11:52:14.549728 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.549771 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.551771 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.552394 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.552401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.552527 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:14.553275 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.553301 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.554337 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.554343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.555547 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.555558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.556367 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.556375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.556606 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.557307 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:14.557310 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:14.557347 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:14.557420 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:14.557787 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:14.557799 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:14.557809 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.557991 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.558370 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
I20260924 11:52:14.560674 21068 heartbeater.cc:347] Connected to a master server at 127.1.210.62:46065
I20260924 11:52:14.561012 21068 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:14.561748 21068 heartbeater.cc:511] Master 127.1.210.62:46065 requested a full tablet report, sending...
I20260924 11:52:14.563174 20621 ts_manager.cc:194] Registered new tserver with Master: 5593f0ab771949d297f3485ecca72bd9 (127.1.210.3:38277)
I20260924 11:52:14.564262 20621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.3:39115
I20260924 11:52:14.565191 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:52:14.575779 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.569711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.1.210.62:46065)
0924 11:52:14.569932 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.569939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.570331 (+ 392us) server_negotiation.cc:407] Connection header received
0924 11:52:14.570528 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.570533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.570618 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.570729 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:14.571196 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.571874 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.573331 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.573756 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.573905 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.575007 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:14.575018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:14.575021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:14.575038 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:14.575067 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:14.575079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:14.575243 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.575442 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.575496 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:52:14.579504 20620 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:52:14.586517 20620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:52:14.621004 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.599120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:44031 (local address 127.0.0.1:56714)
0924 11:52:14.611903 (+ 12783us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.611920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.612037 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.612965 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.612972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.612999 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:14.613464 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.613480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.615767 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.615772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.618251 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.618263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.620023 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.620030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.620230 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.620279 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.620394 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.620587 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12650,"mutex_wait_us":83,"thread_start_us":11061,"threads_started":1}
I20260924 11:52:14.621006 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.599477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.1.210.2:44031)
0924 11:52:14.609773 (+ 10296us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.609786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.612187 (+ 2401us) server_negotiation.cc:407] Connection header received
0924 11:52:14.612287 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.612296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.612511 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.612689 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:14.613686 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.615609 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.618428 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.619816 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.619951 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.620207 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.620485 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.620662 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10035,"thread_start_us":10302,"threads_started":1}
I20260924 11:52:14.621721 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.599077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.1.210.1:42441)
0924 11:52:14.611550 (+ 12473us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.611562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.611598 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:52:14.611723 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.611731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.612030 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.612230 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:14.614333 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.615833 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.618740 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.620087 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.620224 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.620420 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.621198 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.621371 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12333,"thread_start_us":12208,"threads_started":1}
I20260924 11:52:14.622119 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.598868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:42441 (local address 127.0.0.1:37678)
0924 11:52:14.610692 (+ 11824us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.610746 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.610962 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.612511 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.612519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.612554 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:14.614124 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.614138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.615975 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.615983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.618551 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.618566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.620567 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.620574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.620763 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.620801 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.620881 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.621020 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11468,"mutex_wait_us":167,"thread_start_us":12750,"threads_started":1}
I20260924 11:52:14.623138 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.601973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38277 (local address 127.0.0.1:45116)
0924 11:52:14.614993 (+ 13020us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.615015 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.615145 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.616232 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.616239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.616278 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:14.616980 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.616992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.618972 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.618977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.620542 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.620555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.622415 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.622421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.622599 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.622622 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.622689 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.622812 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12877,"mutex_wait_us":112,"thread_start_us":12751,"threads_started":1}
I20260924 11:52:14.625276 20873 tablet_service.cc:1467] Processing CreateTablet for tablet 051b80a8668441c48c3baefa3b1c7744 (DEFAULT_TABLE table=test-table [id=8917ecb1f3d24fae9518433f16466d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:14.625349 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 051b80a8668441c48c3baefa3b1c7744 (DEFAULT_TABLE table=test-table [id=8917ecb1f3d24fae9518433f16466d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:14.626410 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051b80a8668441c48c3baefa3b1c7744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:14.626547 20742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051b80a8668441c48c3baefa3b1c7744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:14.626749 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.602750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.1.210.3:38277)
0924 11:52:14.615624 (+ 12874us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.615630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.615662 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:52:14.615794 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.615804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.616031 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.616218 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:14.617268 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.618841 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.620760 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.622223 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.622350 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.626137 (+ 3787us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.626212 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.626370 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12730,"thread_start_us":12562,"threads_started":1}
I20260924 11:52:14.629722 21003 tablet_service.cc:1467] Processing CreateTablet for tablet 051b80a8668441c48c3baefa3b1c7744 (DEFAULT_TABLE table=test-table [id=8917ecb1f3d24fae9518433f16466d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:14.630752 21003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051b80a8668441c48c3baefa3b1c7744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:14.641693 21086 tablet_bootstrap.cc:492] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Bootstrap starting.
I20260924 11:52:14.643023 21088 tablet_bootstrap.cc:492] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Bootstrap starting.
I20260924 11:52:14.644044 21087 tablet_bootstrap.cc:492] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Bootstrap starting.
I20260924 11:52:14.645823 21086 tablet_bootstrap.cc:654] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:14.646832 21088 tablet_bootstrap.cc:654] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:14.647306 21086 log.cc:824] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:14.647461 21087 tablet_bootstrap.cc:654] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:14.648056 21088 log.cc:824] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:14.648675 21087 log.cc:824] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:14.657125 21086 tablet_bootstrap.cc:492] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: No bootstrap required, opened a new log
I20260924 11:52:14.657454 21086 ts_tablet_manager.cc:1397] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260924 11:52:14.658351 21088 tablet_bootstrap.cc:492] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: No bootstrap required, opened a new log
I20260924 11:52:14.658754 21088 ts_tablet_manager.cc:1397] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 11:52:14.665920 21087 tablet_bootstrap.cc:492] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: No bootstrap required, opened a new log
I20260924 11:52:14.666203 21087 ts_tablet_manager.cc:1397] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.004s
I20260924 11:52:14.667910 21086 raft_consensus.cc:348] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.668278 21086 raft_consensus.cc:374] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:14.668427 21086 raft_consensus.cc:729] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049dfd097b764214b97f3e05c9aa4296, State: Initialized, Role: FOLLOWER
I20260924 11:52:14.669126 21086 consensus_queue.cc:260] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.670208 20807 heartbeater.cc:503] Master 127.1.210.62:46065 was elected leader, sending a full tablet report...
I20260924 11:52:14.670375 21088 raft_consensus.cc:348] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.670797 21088 raft_consensus.cc:374] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:14.670888 21088 raft_consensus.cc:729] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133d2bcbdeaf4eb7866410180a3ebf41, State: Initialized, Role: FOLLOWER
I20260924 11:52:14.671595 21088 consensus_queue.cc:260] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.672215 21086 ts_tablet_manager.cc:1428] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260924 11:52:14.673159 20938 heartbeater.cc:503] Master 127.1.210.62:46065 was elected leader, sending a full tablet report...
I20260924 11:52:14.673715 21088 ts_tablet_manager.cc:1428] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260924 11:52:14.676515 21087 raft_consensus.cc:348] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.677011 21087 raft_consensus.cc:374] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:14.677129 21087 raft_consensus.cc:729] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5593f0ab771949d297f3485ecca72bd9, State: Initialized, Role: FOLLOWER
I20260924 11:52:14.677752 21087 consensus_queue.cc:260] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.678764 21068 heartbeater.cc:503] Master 127.1.210.62:46065 was elected leader, sending a full tablet report...
I20260924 11:52:14.679302 21087 ts_tablet_manager.cc:1428] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 11:52:14.788003 21092 raft_consensus.cc:482] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:52:14.788442 21092 raft_consensus.cc:504] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.790172 21092 leader_election.cc:288] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 133d2bcbdeaf4eb7866410180a3ebf41 (127.1.210.2:44031), 5593f0ab771949d297f3485ecca72bd9 (127.1.210.3:38277)
I20260924 11:52:14.797302 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.790644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:35205 (local address 127.1.210.3:38277)
0924 11:52:14.790869 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.790875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.791663 (+ 788us) server_negotiation.cc:407] Connection header received
0924 11:52:14.791935 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.791941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.792037 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.792124 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:14.792664 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.793494 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.795455 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.796339 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.796435 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.796508 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.796864 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.796992 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:52:14.797420 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.790597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:38277 (local address 127.1.210.1:35205)
0924 11:52:14.791509 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.791538 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.791719 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.792158 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.792162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.792184 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:14.792508 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.792517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.793627 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.793632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.795279 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.795291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.796518 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.796524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.796691 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.796728 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.796820 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.796953 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"spinlock_wait_cycles":9600,"thread_start_us":564,"threads_started":1}
I20260924 11:52:14.797730 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.790732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:39421 (local address 127.1.210.2:44031)
0924 11:52:14.790912 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.790920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.791673 (+ 753us) server_negotiation.cc:407] Connection header received
0924 11:52:14.791905 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.791911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.791992 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.792105 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:14.792922 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.793777 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.795705 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.796726 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.796850 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.796934 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.797341 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.797486 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:52:14.797384 21069 tablet.cc:2369] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:14.798342 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.790611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:44031 (local address 127.1.210.1:39421)
0924 11:52:14.791534 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:14.791549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:14.791719 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:14.792293 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:14.792300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:14.792331 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:14.792731 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.792745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.793940 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.793944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.795526 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:14.795541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.796926 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.796932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:14.797131 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.797163 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:52:14.797939 (+ 776us) client_negotiation.cc:240] Negotiation successful
0924 11:52:14.798035 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":3,"spinlock_wait_cycles":14464,"thread_start_us":501,"threads_started":1}
I20260924 11:52:14.798678 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051b80a8668441c48c3baefa3b1c7744" candidate_uuid: "049dfd097b764214b97f3e05c9aa4296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5593f0ab771949d297f3485ecca72bd9" is_pre_election: true
I20260924 11:52:14.799245 21023 raft_consensus.cc:2454] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 049dfd097b764214b97f3e05c9aa4296 in term 0.
I20260924 11:52:14.799348 20893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051b80a8668441c48c3baefa3b1c7744" candidate_uuid: "049dfd097b764214b97f3e05c9aa4296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" is_pre_election: true
I20260924 11:52:14.799872 20893 raft_consensus.cc:2454] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 049dfd097b764214b97f3e05c9aa4296 in term 0.
I20260924 11:52:14.799901 20698 leader_election.cc:302] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049dfd097b764214b97f3e05c9aa4296, 5593f0ab771949d297f3485ecca72bd9; no voters:
I20260924 11:52:14.800344 21092 raft_consensus.cc:2781] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:52:14.800513 21092 raft_consensus.cc:482] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:52:14.800622 21092 raft_consensus.cc:3037] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:14.803200 21092 raft_consensus.cc:504] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.803988 21092 leader_election.cc:288] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [CANDIDATE]: Term 1 election: Requested vote from peers 133d2bcbdeaf4eb7866410180a3ebf41 (127.1.210.2:44031), 5593f0ab771949d297f3485ecca72bd9 (127.1.210.3:38277)
I20260924 11:52:14.804313 20893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051b80a8668441c48c3baefa3b1c7744" candidate_uuid: "049dfd097b764214b97f3e05c9aa4296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133d2bcbdeaf4eb7866410180a3ebf41"
I20260924 11:52:14.804328 21023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051b80a8668441c48c3baefa3b1c7744" candidate_uuid: "049dfd097b764214b97f3e05c9aa4296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5593f0ab771949d297f3485ecca72bd9"
I20260924 11:52:14.804490 21023 raft_consensus.cc:3037] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:14.804520 20893 raft_consensus.cc:3037] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:14.806844 20893 raft_consensus.cc:2454] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 049dfd097b764214b97f3e05c9aa4296 in term 1.
I20260924 11:52:14.806843 21023 raft_consensus.cc:2454] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 049dfd097b764214b97f3e05c9aa4296 in term 1.
I20260924 11:52:14.807263 20698 leader_election.cc:302] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049dfd097b764214b97f3e05c9aa4296, 5593f0ab771949d297f3485ecca72bd9; no voters:
I20260924 11:52:14.807541 21092 raft_consensus.cc:2781] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:14.807953 21092 raft_consensus.cc:686] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 1 LEADER]: Becoming Leader. State: Replica: 049dfd097b764214b97f3e05c9aa4296, State: Running, Role: LEADER
I20260924 11:52:14.808424 21092 consensus_queue.cc:237] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } }
I20260924 11:52:14.813367 20621 catalog_manager.cc:5495] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 049dfd097b764214b97f3e05c9aa4296 (127.1.210.1). New cstate: current_term: 1 leader_uuid: "049dfd097b764214b97f3e05c9aa4296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049dfd097b764214b97f3e05c9aa4296" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 } health_report { overall_health: UNKNOWN } } }
W20260924 11:52:14.852798 20939 tablet.cc:2369] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:14.861366 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.856323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.1.210.1:42441)
0924 11:52:14.856515 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.856520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.856538 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:52:14.856624 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.856628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.856719 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.856843 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:52:14.857283 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.857963 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.859320 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.859719 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.859812 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.860264 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:52:14.860755 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:52:14.860881 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.860980 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.861015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:52:14.869674 20893 raft_consensus.cc:1261] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 1 FOLLOWER]: Refusing update from remote peer 049dfd097b764214b97f3e05c9aa4296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:14.869660 21023 raft_consensus.cc:1261] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 1 FOLLOWER]: Refusing update from remote peer 049dfd097b764214b97f3e05c9aa4296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:14.870776 21097 consensus_queue.cc:1035] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d2bcbdeaf4eb7866410180a3ebf41" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 44031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:14.871131 21092 consensus_queue.cc:1035] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5593f0ab771949d297f3485ecca72bd9" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 38277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:14.884318 21101 mvcc.cc:205] Tried to move back new op lower bound from 7332867010014846976 to 7332867009779372032. Current Snapshot: MvccSnapshot[applied={T|T < 7332867010014846976}]
I20260924 11:52:14.894593 20620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:14.894943 20620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:14.898608 20620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bffde95f162746829a6215a407d08b62: Sending DeleteTablet for 3 replicas of tablet 051b80a8668441c48c3baefa3b1c7744
I20260924 11:52:14.899802 20742 tablet_service.cc:1514] Processing DeleteTablet for tablet 051b80a8668441c48c3baefa3b1c7744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:14 UTC) from {username='slave'} at 127.0.0.1:37678
I20260924 11:52:14.900561 20873 tablet_service.cc:1514] Processing DeleteTablet for tablet 051b80a8668441c48c3baefa3b1c7744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:14 UTC) from {username='slave'} at 127.0.0.1:56714
I20260924 11:52:14.901110 21109 tablet_replica.cc:330] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: stopping tablet replica
I20260924 11:52:14.901499 21110 tablet_replica.cc:330] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: stopping tablet replica
I20260924 11:52:14.901808 21003 tablet_service.cc:1514] Processing DeleteTablet for tablet 051b80a8668441c48c3baefa3b1c7744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:14 UTC) from {username='slave'} at 127.0.0.1:45116
I20260924 11:52:14.901882 21110 raft_consensus.cc:2229] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:14.902441 21110 raft_consensus.cc:2258] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:14.902508 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20680
I20260924 11:52:14.903971 21109 raft_consensus.cc:2229] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:52:14.904794 21109 raft_consensus.cc:2258] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:14.904896 21110 ts_tablet_manager.cc:1905] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:14.914110 21110 ts_tablet_manager.cc:1918] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:14.914369 21110 log.cc:1196] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/051b80a8668441c48c3baefa3b1c7744
I20260924 11:52:14.914975 21110 ts_tablet_manager.cc:1939] T 051b80a8668441c48c3baefa3b1c7744 P 133d2bcbdeaf4eb7866410180a3ebf41: Deleting consensus metadata
I20260924 11:52:14.916954 20606 catalog_manager.cc:4841] TS 133d2bcbdeaf4eb7866410180a3ebf41 (127.1.210.2:44031): tablet 051b80a8668441c48c3baefa3b1c7744 (table test-table [id=8917ecb1f3d24fae9518433f16466d45]) successfully deleted
I20260924 11:52:14.919193 21109 ts_tablet_manager.cc:1905] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:14.919430 21111 tablet_replica.cc:330] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: stopping tablet replica
I20260924 11:52:14.919907 21111 raft_consensus.cc:2229] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:14.920449 21111 raft_consensus.cc:2258] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:14.922185 21111 ts_tablet_manager.cc:1905] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:14.924981 21109 ts_tablet_manager.cc:1918] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:14.925202 21109 log.cc:1196] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/051b80a8668441c48c3baefa3b1c7744
I20260924 11:52:14.925746 21109 ts_tablet_manager.cc:1939] T 051b80a8668441c48c3baefa3b1c7744 P 049dfd097b764214b97f3e05c9aa4296: Deleting consensus metadata
I20260924 11:52:14.926407 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:14.904747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.1.210.1:42441)
0924 11:52:14.904969 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:14.904975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:14.915078 (+ 10103us) server_negotiation.cc:407] Connection header received
0924 11:52:14.915573 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:14.915579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:14.915703 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:14.915781 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:14.916437 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.917597 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.918941 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:14.919601 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:14.919726 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:14.924599 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:14.924610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:14.924627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:14.925486 (+ 859us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:14.925549 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:14.925575 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:14.925694 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:14.925925 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 11:52:14.926045 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:52:14.927536 20605 catalog_manager.cc:4841] TS 049dfd097b764214b97f3e05c9aa4296 (127.1.210.1:42441): tablet 051b80a8668441c48c3baefa3b1c7744 (table test-table [id=8917ecb1f3d24fae9518433f16466d45]) successfully deleted
I20260924 11:52:14.929191 21111 ts_tablet_manager.cc:1918] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:14.929354 21111 log.cc:1196] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/051b80a8668441c48c3baefa3b1c7744
I20260924 11:52:14.929904 21111 ts_tablet_manager.cc:1939] T 051b80a8668441c48c3baefa3b1c7744 P 5593f0ab771949d297f3485ecca72bd9: Deleting consensus metadata
I20260924 11:52:14.933142 20802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:14.934736 20608 catalog_manager.cc:4841] TS 5593f0ab771949d297f3485ecca72bd9 (127.1.210.3:38277): tablet 051b80a8668441c48c3baefa3b1c7744 (table test-table [id=8917ecb1f3d24fae9518433f16466d45]) successfully deleted
I20260924 11:52:15.039491 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20680
I20260924 11:52:15.058133 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20811
I20260924 11:52:15.064882 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:15.058933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.1.210.2:44031)
0924 11:52:15.059120 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:15.059125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:15.059249 (+ 124us) server_negotiation.cc:407] Connection header received
0924 11:52:15.059431 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:15.059435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:15.059510 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:15.059602 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:15.060064 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:15.060766 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:15.062155 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:15.062533 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:15.062624 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:15.063574 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:15.063582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:15.063597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:15.064136 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:15.064191 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:15.064211 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:15.064318 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:15.064435 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 11:52:15.064514 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:52:15.065568 20933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:15.155714 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20811
I20260924 11:52:15.172232 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20941
I20260924 11:52:15.178762 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:15.172993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.1.210.3:38277)
0924 11:52:15.173196 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:15.173200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:15.173219 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:52:15.173314 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:15.173318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:15.173400 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:15.173484 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:15.173863 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:15.174583 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:15.175812 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:15.176199 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:15.176289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:15.177437 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:15.177447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:15.177462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:15.178021 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:15.178069 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:15.178089 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:15.178235 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:15.178356 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 11:52:15.178464 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:52:15.179438 21063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:15.267172 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20941
I20260924 11:52:15.283746 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 20590
I20260924 11:52:15.284819 20651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:15.393447 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 20590
2026-09-24T11:52:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2850 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:15 dist-test-slave-dd32 krb5kdc[21119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:15 dist-test-slave-dd32 krb5kdc[21119](info): set up 2 sockets
Sep 24 11:52:15 dist-test-slave-dd32 krb5kdc[21119](info): commencing operation
krb5kdc: starting...
W20260924 11:52:16.561542 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:16 dist-test-slave-dd32 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:16.812886 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37243
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:17.145115 21132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:17.145428 21132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:17.145519 21132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:17.152885 21132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:17.153000 21132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:17.153047 21132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:17.153196 21132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:17.153249 21132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:17.164390 21132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:37243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:17.165750 21132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:17.166983 21132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:17.178659 21138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:17.178625 21137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:17.179108 21140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:17.179555 21132 server_base.cc:1034] running on GCE node
I20260924 11:52:17.180285 21132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:17.181671 21132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:17.182906 21132 hybrid_clock.cc:648] HybridClock initialized: now 1790250737182871 us; error 54 us; skew 500 ppm
I20260924 11:52:17.183254 21132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:17 dist-test-slave-dd32 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:17.186631 21132 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:17.188057 21132 webserver.cc:460] Webserver started at http://127.1.210.62:38519/ using document root <none> and password file <none>
I20260924 11:52:17.188763 21132 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:17.188920 21132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:17.189311 21132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:17.192381 21132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "30ffd12654ee4adda9feed0ec56b2258"
format_stamp: "Formatted at 2026-09-24 11:52:17 on dist-test-slave-dd32"
server_key: "61a0bd5ff11e76c8b2fb538e1e1cd501"
server_key_iv: "7ae324e47f34b938e1d503ac2d6dec05"
server_key_version: "encryptionkey@0"
I20260924 11:52:17.193244 21132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "30ffd12654ee4adda9feed0ec56b2258"
format_stamp: "Formatted at 2026-09-24 11:52:17 on dist-test-slave-dd32"
server_key: "61a0bd5ff11e76c8b2fb538e1e1cd501"
server_key_iv: "7ae324e47f34b938e1d503ac2d6dec05"
server_key_version: "encryptionkey@0"
I20260924 11:52:17.198393 21132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 11:52:17.201849 21147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:17.203066 21132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:17.203246 21132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "30ffd12654ee4adda9feed0ec56b2258"
format_stamp: "Formatted at 2026-09-24 11:52:17 on dist-test-slave-dd32"
server_key: "61a0bd5ff11e76c8b2fb538e1e1cd501"
server_key_iv: "7ae324e47f34b938e1d503ac2d6dec05"
server_key_version: "encryptionkey@0"
I20260924 11:52:17.203446 21132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:17.228570 21132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:17.232398 21132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:17.232725 21132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:17.250875 21132 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:37243
I20260924 11:52:17.250913 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:37243 every 8 connection(s)
I20260924 11:52:17.252235 21132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:17.255836 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:17.256587 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21132
I20260924 11:52:17.256759 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:17.257289 1864 external_mini_cluster.cc:1428] Setting key 4b8a9775db345ce298d179a43436ff2b
I20260924 11:52:17.270179 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258: Bootstrap starting.
Sep 24 11:52:17 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:17.273169 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:17.274549 21199 log.cc:824] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:17.277899 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258: No bootstrap required, opened a new log
I20260924 11:52:17.281862 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:17.259255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.1.210.62:37243)
0924 11:52:17.260083 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:17.260098 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:17.260146 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:52:17.262618 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:17.262647 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:17.263149 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:17.263604 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:17.264925 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:17.266921 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:17.268315 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:17.269391 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:17.269580 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:17.273523 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:17.273535 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:17.273558 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:17.273614 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:17.277519 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:17.278192 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:17.278199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:17.278206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:17.278447 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:17.278807 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:17.278812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:17.278815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:17.279452 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:17.279781 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:17.280055 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 11:52:17.280507 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":378,"threads_started":1}
I20260924 11:52:17.287468 21199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffd12654ee4adda9feed0ec56b2258" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37243 } }
I20260924 11:52:17.287792 21199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:17.287865 21199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ffd12654ee4adda9feed0ec56b2258, State: Initialized, Role: FOLLOWER
I20260924 11:52:17.288383 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffd12654ee4adda9feed0ec56b2258" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37243 } }
I20260924 11:52:17.288581 21199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:17.288684 21199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:17.288820 21199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:17.291172 21199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffd12654ee4adda9feed0ec56b2258" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37243 } }
I20260924 11:52:17.291584 21199 leader_election.cc:302] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ffd12654ee4adda9feed0ec56b2258; no voters:
I20260924 11:52:17.292096 21199 leader_election.cc:288] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:17.292209 21204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:17.292898 21204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [term 1 LEADER]: Becoming Leader. State: Replica: 30ffd12654ee4adda9feed0ec56b2258, State: Running, Role: LEADER
I20260924 11:52:17.293452 21204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffd12654ee4adda9feed0ec56b2258" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37243 } }
I20260924 11:52:17.294196 21199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:17.297015 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ffd12654ee4adda9feed0ec56b2258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffd12654ee4adda9feed0ec56b2258" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37243 } } }
I20260924 11:52:17.297461 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:17.300467 21206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ffd12654ee4adda9feed0ec56b2258. Latest consensus state: current_term: 1 leader_uuid: "30ffd12654ee4adda9feed0ec56b2258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffd12654ee4adda9feed0ec56b2258" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 37243 } } }
I20260924 11:52:17.300740 21206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:17.301728 21213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:17.305954 21213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:17.313295 21213 catalog_manager.cc:1347] Generated new cluster ID: 6f8df127a4e6425f9b6c48d347b06240
I20260924 11:52:17.313402 21213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:17.349910 21213 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:17.351225 21213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:17.360096 21213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258: Generated new TSK 0
I20260924 11:52:17.360940 21213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
Entry for principal kudu/127.1.210.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab.
I20260924 11:52:17.433384 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37243
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:17.780120 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:17.780392 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:17.780570 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:17.787767 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:17.787864 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:17.788013 21225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:52:17.797705 21225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.1.keytab
--principal=kudu/127.1.210.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:17.799013 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:17.800335 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:17.813973 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:17.814062 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:17.814096 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:17.814980 21225 server_base.cc:1034] running on GCE node
I20260924 11:52:17.815579 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:17.816480 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:17.817770 21225 hybrid_clock.cc:648] HybridClock initialized: now 1790250737817673 us; error 49 us; skew 500 ppm
I20260924 11:52:17.818094 21225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:17 dist-test-slave-dd32 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:17.821184 21225 init.cc:377] Logged in from keytab as kudu/127.1.210.1@KRBTEST.COM (short username kudu)
I20260924 11:52:17.822801 21225 webserver.cc:460] Webserver started at http://127.1.210.1:38527/ using document root <none> and password file <none>
I20260924 11:52:17.823482 21225 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:17.823619 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:17.823917 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:17.826857 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "2ad6ead3587f4372b2a308ddf9445a05"
format_stamp: "Formatted at 2026-09-24 11:52:17 on dist-test-slave-dd32"
server_key: "9a61d0dfcffd8b3c5429227fc14c8287"
server_key_iv: "10c474dbda4a111a5329e498123579fe"
server_key_version: "encryptionkey@0"
I20260924 11:52:17.827625 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "2ad6ead3587f4372b2a308ddf9445a05"
format_stamp: "Formatted at 2026-09-24 11:52:17 on dist-test-slave-dd32"
server_key: "9a61d0dfcffd8b3c5429227fc14c8287"
server_key_iv: "10c474dbda4a111a5329e498123579fe"
server_key_version: "encryptionkey@0"
I20260924 11:52:17.832543 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 11:52:17.835898 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:17.837266 21225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:52:17.837427 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "2ad6ead3587f4372b2a308ddf9445a05"
format_stamp: "Formatted at 2026-09-24 11:52:17 on dist-test-slave-dd32"
server_key: "9a61d0dfcffd8b3c5429227fc14c8287"
server_key_iv: "10c474dbda4a111a5329e498123579fe"
server_key_version: "encryptionkey@0"
I20260924 11:52:17.837631 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:17.860710 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:17.864665 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:17.864956 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:17.865906 21225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:17.867195 21225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:17.867298 21225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:17.867446 21225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:17.867522 21225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:17.900453 21225 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:37421
I20260924 11:52:17.900503 21352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:37421 every 8 connection(s)
I20260924 11:52:17.901787 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:52:17.911969 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21225
I20260924 11:52:17.912146 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:52:17.912695 1864 external_mini_cluster.cc:1428] Setting key b04bfaf5e5d7a1167e030855eb66a8ad
Sep 24 11:52:17 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.1@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:17.922947 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:17.904288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:33727 (local address 127.1.210.62:37243)
0924 11:52:17.904743 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:17.904748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:17.905751 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 11:52:17.907615 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:17.907621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:17.907717 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:17.907829 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:17.909350 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:17.910082 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:17.911574 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:17.912221 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:17.912361 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:17.916089 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:17.916099 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:17.916107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:17.916154 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:17.919630 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:17.920379 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:17.920387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:17.920393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:17.920553 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:17.920953 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:17.920961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:17.920967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:17.921371 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:17.921524 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:17.922206 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 11:52:17.922466 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":253,"threads_started":1}
I20260924 11:52:17.923538 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:17.904866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37243 (local address 127.1.210.1:33727)
0924 11:52:17.905607 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:17.905643 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:17.907346 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:17.908058 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:17.908066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:17.908572 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:17.909183 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:17.909203 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:17.910228 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:17.910233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:17.911425 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:17.911437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:17.912435 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:17.912441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:17.912644 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:17.913412 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:17.913442 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:17.915875 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:17.919782 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:17.919789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:17.919801 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:17.920247 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:17.920656 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:17.920659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:17.920661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:17.920835 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:17.921493 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:17.921499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:17.921812 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 11:52:17.922033 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 11:52:17.922424 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":322,"threads_started":1}
I20260924 11:52:17.925256 21353 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37243
I20260924 11:52:17.925580 21353 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:17.926299 21353 heartbeater.cc:511] Master 127.1.210.62:37243 requested a full tablet report, sending...
I20260924 11:52:17.928658 21164 ts_manager.cc:194] Registered new tserver with Master: 2ad6ead3587f4372b2a308ddf9445a05 (127.1.210.1:37421)
I20260924 11:52:17.930876 21164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.1@KRBTEST.COM'} at 127.1.210.1:33727
WARNING: no policy specified for kudu/127.1.210.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
Entry for principal kudu/127.1.210.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab.
I20260924 11:52:17.969965 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37243
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:18.295356 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:18.295639 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:18.295837 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:18.302989 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:18.303098 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:18.303284 21359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:52:18.312937 21359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.2.keytab
--principal=kudu/127.1.210.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:18.314213 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:18.315483 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:18.328372 21365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:18.328341 21364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:18.328501 21367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:18.329051 21359 server_base.cc:1034] running on GCE node
I20260924 11:52:18.329785 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:18.330737 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:18.331969 21359 hybrid_clock.cc:648] HybridClock initialized: now 1790250738331903 us; error 55 us; skew 500 ppm
I20260924 11:52:18.332567 21359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:18 dist-test-slave-dd32 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:18.335898 21359 init.cc:377] Logged in from keytab as kudu/127.1.210.2@KRBTEST.COM (short username kudu)
I20260924 11:52:18.337412 21359 webserver.cc:460] Webserver started at http://127.1.210.2:36463/ using document root <none> and password file <none>
I20260924 11:52:18.338222 21359 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:18.338383 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:18.338743 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:18.341866 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "0fb560268a014101a9f7d16d308c9931"
format_stamp: "Formatted at 2026-09-24 11:52:18 on dist-test-slave-dd32"
server_key: "b31494a4824525cf260453cd350e952d"
server_key_iv: "22c968f2008f2451e7b8577bc4578fd1"
server_key_version: "encryptionkey@0"
I20260924 11:52:18.342811 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "0fb560268a014101a9f7d16d308c9931"
format_stamp: "Formatted at 2026-09-24 11:52:18 on dist-test-slave-dd32"
server_key: "b31494a4824525cf260453cd350e952d"
server_key_iv: "22c968f2008f2451e7b8577bc4578fd1"
server_key_version: "encryptionkey@0"
I20260924 11:52:18.348136 21359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:52:18.351917 21374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:18.353688 21359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:18.353853 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "0fb560268a014101a9f7d16d308c9931"
format_stamp: "Formatted at 2026-09-24 11:52:18 on dist-test-slave-dd32"
server_key: "b31494a4824525cf260453cd350e952d"
server_key_iv: "22c968f2008f2451e7b8577bc4578fd1"
server_key_version: "encryptionkey@0"
I20260924 11:52:18.354027 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:18.382558 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:18.386593 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:18.386871 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:18.387830 21359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:18.389106 21359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:18.389176 21359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:18.389341 21359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:18.389391 21359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:18.422833 21359 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:46409
I20260924 11:52:18.422883 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:46409 every 8 connection(s)
I20260924 11:52:18.424659 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:52:18.434692 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21359
I20260924 11:52:18.434854 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:52:18.435272 1864 external_mini_cluster.cc:1428] Setting key 993ebe8ea86f0fe50c2e79e71f24bf07
Sep 24 11:52:18 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.2@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:18.447083 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:18.427384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:36135 (local address 127.1.210.62:37243)
0924 11:52:18.427818 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:18.427824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:18.428883 (+ 1059us) server_negotiation.cc:407] Connection header received
0924 11:52:18.431281 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:18.431286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:18.431353 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:18.431450 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:18.432925 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.433705 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.435445 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.435993 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.436158 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:18.440212 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:18.440217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:18.440221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:18.440253 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:18.443831 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:18.444553 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:18.444561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:18.444568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:18.444730 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:18.445201 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:18.445209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:18.445215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:18.445602 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:18.445759 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:18.446653 (+ 894us) server_negotiation.cc:299] Negotiation successful
0924 11:52:18.446823 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":213,"threads_started":1}
I20260924 11:52:18.448679 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:18.427974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37243 (local address 127.1.210.2:36135)
0924 11:52:18.428719 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:18.428766 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:18.431003 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:18.431687 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:18.431695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:18.432224 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:18.432779 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:18.432795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.433838 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.433842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:18.435274 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:18.435286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.436237 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.436245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:18.436478 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:18.437234 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:18.437265 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:18.439958 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:18.443967 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:18.443975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:18.443989 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:18.444430 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:18.444841 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:18.444844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:18.444846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:18.445035 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:18.445723 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:18.445728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:18.446228 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 11:52:18.446551 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 11:52:18.447192 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":283,"threads_started":1}
I20260924 11:52:18.450259 21487 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37243
I20260924 11:52:18.450572 21487 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:18.451310 21487 heartbeater.cc:511] Master 127.1.210.62:37243 requested a full tablet report, sending...
I20260924 11:52:18.452864 21164 ts_manager.cc:194] Registered new tserver with Master: 0fb560268a014101a9f7d16d308c9931 (127.1.210.2:46409)
I20260924 11:52:18.454174 21164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.2@KRBTEST.COM'} at 127.1.210.2:36135
WARNING: no policy specified for kudu/127.1.210.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
Entry for principal kudu/127.1.210.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab.
I20260924 11:52:18.491942 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37243
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:18.813887 21493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:18.814148 21493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:18.814247 21493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:18.823352 21493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:18.823491 21493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:18.823693 21493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:52:18.835371 21493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.3.keytab
--principal=kudu/127.1.210.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:37243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:18.836588 21493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:18.838065 21493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:18.851213 21499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:18.851243 21501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:18.851605 21498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:18.851873 21493 server_base.cc:1034] running on GCE node
I20260924 11:52:18.852603 21493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:18.853566 21493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:18.854768 21493 hybrid_clock.cc:648] HybridClock initialized: now 1790250738854727 us; error 26 us; skew 500 ppm
I20260924 11:52:18.855064 21493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:18 dist-test-slave-dd32 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:18.858186 21493 init.cc:377] Logged in from keytab as kudu/127.1.210.3@KRBTEST.COM (short username kudu)
I20260924 11:52:18.859496 21493 webserver.cc:460] Webserver started at http://127.1.210.3:38147/ using document root <none> and password file <none>
I20260924 11:52:18.860146 21493 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:18.860289 21493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:18.860587 21493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:18.863390 21493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "ca30cd7559d6419291364a62318b29db"
format_stamp: "Formatted at 2026-09-24 11:52:18 on dist-test-slave-dd32"
server_key: "5b24149c33be0cd29e3242c311eb0929"
server_key_iv: "cec51872f82d2966f865d696e89fb6f5"
server_key_version: "encryptionkey@0"
I20260924 11:52:18.864223 21493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "ca30cd7559d6419291364a62318b29db"
format_stamp: "Formatted at 2026-09-24 11:52:18 on dist-test-slave-dd32"
server_key: "5b24149c33be0cd29e3242c311eb0929"
server_key_iv: "cec51872f82d2966f865d696e89fb6f5"
server_key_version: "encryptionkey@0"
I20260924 11:52:18.869241 21493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:52:18.872474 21508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:18.873742 21493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:18.873908 21493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "ca30cd7559d6419291364a62318b29db"
format_stamp: "Formatted at 2026-09-24 11:52:18 on dist-test-slave-dd32"
server_key: "5b24149c33be0cd29e3242c311eb0929"
server_key_iv: "cec51872f82d2966f865d696e89fb6f5"
server_key_version: "encryptionkey@0"
I20260924 11:52:18.874112 21493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:18.892043 21493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:18.896044 21493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:18.896282 21493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:18.897235 21493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:18.898566 21493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:18.898638 21493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:18.898762 21493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:18.898813 21493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:18.934276 21353 heartbeater.cc:503] Master 127.1.210.62:37243 was elected leader, sending a full tablet report...
I20260924 11:52:18.938069 21493 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:40573
I20260924 11:52:18.938202 21620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:40573 every 8 connection(s)
I20260924 11:52:18.939720 21493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:52:18.946161 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21493
I20260924 11:52:18.946334 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:52:18.946869 1864 external_mini_cluster.cc:1428] Setting key 710e3eb6199426f8b41868e93bc12303
Sep 24 11:52:18 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.3@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:18.961961 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:18.945335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:50507 (local address 127.1.210.62:37243)
0924 11:52:18.945511 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:18.945516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:18.945540 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:52:18.948063 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:18.948070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:18.948157 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:18.948229 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:18.950540 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.951303 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.952822 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.953286 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.953417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:18.956621 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:18.956626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:18.956629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:18.956662 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:18.959238 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:18.959893 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:18.959897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:18.959900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:18.960012 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:18.960366 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:18.960369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:18.960371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:18.960637 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:18.960749 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:18.961482 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 11:52:18.961681 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:52:18.963248 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:18.943324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:37243 (local address 127.1.210.3:50507)
0924 11:52:18.944546 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:18.944593 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:18.947736 (+ 3143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:18.948766 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:18.948778 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:18.949579 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:18.950382 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:18.950401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.951434 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.951437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:18.952620 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:18.952632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.953504 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.953509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:18.953755 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:18.954353 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:18.954379 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:18.956438 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:18.959360 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:18.959366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:18.959379 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:18.959796 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:18.960102 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:18.960105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:18.960107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:18.960276 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:18.960745 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:18.960750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:18.961109 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 11:52:18.961384 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 11:52:18.961901 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":3952,"threads_started":1}
I20260924 11:52:18.964653 21621 heartbeater.cc:347] Connected to a master server at 127.1.210.62:37243
I20260924 11:52:18.964962 21621 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:18.965714 21621 heartbeater.cc:511] Master 127.1.210.62:37243 requested a full tablet report, sending...
I20260924 11:52:18.967114 21164 ts_manager.cc:194] Registered new tserver with Master: ca30cd7559d6419291364a62318b29db (127.1.210.3:40573)
I20260924 11:52:18.967934 21164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.3@KRBTEST.COM'} at 127.1.210.3:50507
I20260924 11:52:18.976603 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:52:18.993777 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:18.980761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.1.210.62:37243)
0924 11:52:18.980929 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:18.980934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:18.981460 (+ 526us) server_negotiation.cc:407] Connection header received
0924 11:52:18.981666 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:18.981671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:18.981748 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:18.981858 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:18.983235 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.983970 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.985345 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:18.985750 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:18.985918 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:18.988438 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:18.988443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:18.988447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:18.988478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:18.991045 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:18.991746 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:18.991756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:18.991759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:18.991909 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:18.992281 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:18.992287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:18.992290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:18.992605 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:18.992883 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:18.993223 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 11:52:18.993438 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:52:18.997611 21164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:52:19.004572 21164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:52:19.030324 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.017170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:40573 (local address 127.0.0.1:58214)
0924 11:52:19.017944 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:19.017988 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:19.018158 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:19.019681 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:19.019691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:19.019737 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:19.020232 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.020249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.022870 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.022879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.026066 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.026087 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.028934 (+ 2847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.028943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.029279 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.029355 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 11:52:19.029521 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 11:52:19.029785 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":349,"threads_started":1}
I20260924 11:52:19.031754 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.017299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.1.210.3:40573)
0924 11:52:19.018594 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.018602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.018634 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:52:19.018734 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.018744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.019062 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.019312 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:19.020486 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.022659 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.026318 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.028286 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.028447 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.028763 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.031034 (+ 2271us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.031295 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":486,"threads_started":1}
I20260924 11:52:19.036849 21556 tablet_service.cc:1467] Processing CreateTablet for tablet 921298eb1b0649be829fedfef306bf5d (DEFAULT_TABLE table=test-table [id=7072133d8e5941f0b72c740fe4308b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:19.038769 21556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921298eb1b0649be829fedfef306bf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:19.046360 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.018420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:46409 (local address 127.0.0.1:57012)
0924 11:52:19.030667 (+ 12247us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:19.030692 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:19.030822 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:19.037356 (+ 6534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:19.037363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:19.037415 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:19.037670 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.037778 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.041418 (+ 3640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.041423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.043483 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.043498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.045619 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.045626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.045828 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.045868 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:52:19.045926 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 11:52:19.046071 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12075,"thread_start_us":12456,"threads_started":1}
I20260924 11:52:19.047523 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.018904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.1.210.2:46409)
0924 11:52:19.036380 (+ 17476us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.036391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.036428 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:52:19.036568 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.036579 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.036861 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.037997 (+ 1136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:19.038260 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.041254 (+ 2994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.043673 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.045176 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.046786 (+ 1610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.046992 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.047053 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.047218 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17258,"thread_start_us":18157,"threads_started":1}
I20260924 11:52:19.051100 21422 tablet_service.cc:1467] Processing CreateTablet for tablet 921298eb1b0649be829fedfef306bf5d (DEFAULT_TABLE table=test-table [id=7072133d8e5941f0b72c740fe4308b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:19.052279 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.018643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:37421 (local address 127.0.0.1:53596)
0924 11:52:19.031130 (+ 12487us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:19.031149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:19.031257 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:19.039836 (+ 8579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:19.039840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:19.039865 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:19.040296 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.040307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.042340 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.042346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.050201 (+ 7855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.050214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.051489 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.051496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.051664 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.051709 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 11:52:19.051796 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:52:19.051915 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12360,"thread_start_us":26736,"threads_started":1}
I20260924 11:52:19.052764 21422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921298eb1b0649be829fedfef306bf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:19.053141 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.018998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.1.210.1:37421)
0924 11:52:19.039415 (+ 20417us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.039421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.039441 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:52:19.039516 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.039520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.039666 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.039861 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:19.040487 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.042193 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.050377 (+ 8184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.051282 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.051441 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.051668 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.052399 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.052574 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20259,"thread_start_us":20091,"threads_started":1}
I20260924 11:52:19.055902 21288 tablet_service.cc:1467] Processing CreateTablet for tablet 921298eb1b0649be829fedfef306bf5d (DEFAULT_TABLE table=test-table [id=7072133d8e5941f0b72c740fe4308b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:19.057274 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921298eb1b0649be829fedfef306bf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:19.057294 21639 tablet_bootstrap.cc:492] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Bootstrap starting.
I20260924 11:52:19.061164 21639 tablet_bootstrap.cc:654] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:19.062459 21639 log.cc:824] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:19.065600 21639 tablet_bootstrap.cc:492] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: No bootstrap required, opened a new log
I20260924 11:52:19.065871 21639 ts_tablet_manager.cc:1397] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:52:19.066114 21641 tablet_bootstrap.cc:492] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Bootstrap starting.
I20260924 11:52:19.069394 21641 tablet_bootstrap.cc:654] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:19.070346 21641 log.cc:824] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:19.073035 21641 tablet_bootstrap.cc:492] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: No bootstrap required, opened a new log
I20260924 11:52:19.073242 21643 tablet_bootstrap.cc:492] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Bootstrap starting.
I20260924 11:52:19.073374 21641 ts_tablet_manager.cc:1397] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 11:52:19.075624 21639 raft_consensus.cc:348] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.075994 21639 raft_consensus.cc:374] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:19.076105 21639 raft_consensus.cc:729] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca30cd7559d6419291364a62318b29db, State: Initialized, Role: FOLLOWER
I20260924 11:52:19.076771 21643 tablet_bootstrap.cc:654] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:19.076711 21639 consensus_queue.cc:260] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.077983 21621 heartbeater.cc:503] Master 127.1.210.62:37243 was elected leader, sending a full tablet report...
I20260924 11:52:19.078070 21643 log.cc:824] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:19.078668 21639 ts_tablet_manager.cc:1428] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260924 11:52:19.080927 21643 tablet_bootstrap.cc:492] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: No bootstrap required, opened a new log
I20260924 11:52:19.081236 21643 ts_tablet_manager.cc:1397] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 11:52:19.084641 21641 raft_consensus.cc:348] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.084972 21641 raft_consensus.cc:374] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:19.085099 21641 raft_consensus.cc:729] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb560268a014101a9f7d16d308c9931, State: Initialized, Role: FOLLOWER
I20260924 11:52:19.085673 21641 consensus_queue.cc:260] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.086694 21487 heartbeater.cc:503] Master 127.1.210.62:37243 was elected leader, sending a full tablet report...
I20260924 11:52:19.087218 21641 ts_tablet_manager.cc:1428] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 11:52:19.090413 21643 raft_consensus.cc:348] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.090847 21643 raft_consensus.cc:374] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:19.090976 21643 raft_consensus.cc:729] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad6ead3587f4372b2a308ddf9445a05, State: Initialized, Role: FOLLOWER
I20260924 11:52:19.091481 21643 consensus_queue.cc:260] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.092953 21643 ts_tablet_manager.cc:1428] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260924 11:52:19.154119 21354 tablet.cc:2369] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:52:19.177196 21488 tablet.cc:2369] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:52:19.196564 21622 tablet.cc:2369] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:19.207322 21646 raft_consensus.cc:482] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:52:19.207643 21646 raft_consensus.cc:504] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.209349 21646 leader_election.cc:288] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca30cd7559d6419291364a62318b29db (127.1.210.3:40573), 2ad6ead3587f4372b2a308ddf9445a05 (127.1.210.1:37421)
I20260924 11:52:19.218335 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.210014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:58877 (local address 127.1.210.1:37421)
0924 11:52:19.210233 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.210241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.211371 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 11:52:19.211547 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.211551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.211610 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.211688 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:19.212206 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.213316 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.215684 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.217039 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.217222 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.217339 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.217900 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.218060 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:52:19.218331 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.209771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:34723 (local address 127.1.210.3:40573)
0924 11:52:19.209979 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.209986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.210760 (+ 774us) server_negotiation.cc:407] Connection header received
0924 11:52:19.210942 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.210952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.211082 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.211201 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:19.211864 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.213027 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.215687 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.217062 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.217222 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.217339 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.217900 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.218055 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:52:19.218535 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.209679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:40573 (local address 127.1.210.2:34723)
0924 11:52:19.210610 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:19.210634 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:19.210791 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:19.211226 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:19.211232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:19.211263 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:19.211693 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.211704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.213186 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.213190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.215479 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.215493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.217330 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.217334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.217606 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.217745 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 11:52:19.217860 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 11:52:19.218077 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":133,"thread_start_us":563,"threads_started":1}
I20260924 11:52:19.218917 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.209791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:37421 (local address 127.1.210.2:58877)
0924 11:52:19.211270 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:19.211287 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:19.211428 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:19.211718 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:19.211722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:19.211750 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:19.212047 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.212055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.213433 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.213441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.215478 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:19.215493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.217327 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.217333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:19.217608 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.217746 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 11:52:19.217854 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:52:19.218077 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1342,"mutex_wait_us":161,"thread_start_us":1206,"threads_started":1}
I20260924 11:52:19.219578 21576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921298eb1b0649be829fedfef306bf5d" candidate_uuid: "0fb560268a014101a9f7d16d308c9931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca30cd7559d6419291364a62318b29db" is_pre_election: true
I20260924 11:52:19.220058 21576 raft_consensus.cc:2454] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb560268a014101a9f7d16d308c9931 in term 0.
I20260924 11:52:19.220043 21308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921298eb1b0649be829fedfef306bf5d" candidate_uuid: "0fb560268a014101a9f7d16d308c9931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad6ead3587f4372b2a308ddf9445a05" is_pre_election: true
I20260924 11:52:19.220670 21308 raft_consensus.cc:2454] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb560268a014101a9f7d16d308c9931 in term 0.
I20260924 11:52:19.220736 21377 leader_election.cc:302] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb560268a014101a9f7d16d308c9931, ca30cd7559d6419291364a62318b29db; no voters:
I20260924 11:52:19.221119 21646 raft_consensus.cc:2781] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:52:19.221272 21646 raft_consensus.cc:482] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:52:19.221390 21646 raft_consensus.cc:3037] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:19.223560 21646 raft_consensus.cc:504] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.224203 21646 leader_election.cc:288] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [CANDIDATE]: Term 1 election: Requested vote from peers ca30cd7559d6419291364a62318b29db (127.1.210.3:40573), 2ad6ead3587f4372b2a308ddf9445a05 (127.1.210.1:37421)
I20260924 11:52:19.224634 21576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921298eb1b0649be829fedfef306bf5d" candidate_uuid: "0fb560268a014101a9f7d16d308c9931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca30cd7559d6419291364a62318b29db"
I20260924 11:52:19.224759 21308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921298eb1b0649be829fedfef306bf5d" candidate_uuid: "0fb560268a014101a9f7d16d308c9931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad6ead3587f4372b2a308ddf9445a05"
I20260924 11:52:19.224856 21576 raft_consensus.cc:3037] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:19.224959 21308 raft_consensus.cc:3037] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:19.228124 21308 raft_consensus.cc:2454] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb560268a014101a9f7d16d308c9931 in term 1.
I20260924 11:52:19.228125 21576 raft_consensus.cc:2454] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb560268a014101a9f7d16d308c9931 in term 1.
I20260924 11:52:19.228549 21378 leader_election.cc:302] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb560268a014101a9f7d16d308c9931, 2ad6ead3587f4372b2a308ddf9445a05; no voters:
I20260924 11:52:19.228853 21646 raft_consensus.cc:2781] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:19.229318 21646 raft_consensus.cc:686] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb560268a014101a9f7d16d308c9931, State: Running, Role: LEADER
I20260924 11:52:19.229773 21646 consensus_queue.cc:237] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } }
I20260924 11:52:19.234416 21163 catalog_manager.cc:5495] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fb560268a014101a9f7d16d308c9931 (127.1.210.2). New cstate: current_term: 1 leader_uuid: "0fb560268a014101a9f7d16d308c9931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fb560268a014101a9f7d16d308c9931" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 } health_report { overall_health: UNKNOWN } } }
I20260924 11:52:19.292712 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.287367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.1.210.2:46409)
0924 11:52:19.287642 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.287646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.287661 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:52:19.287744 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.287747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.287833 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.287984 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:52:19.288357 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.289167 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.290583 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.290992 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.291117 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.291629 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:52:19.292124 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:52:19.292285 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.292336 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.292415 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:52:19.300963 21308 raft_consensus.cc:1261] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 1 FOLLOWER]: Refusing update from remote peer 0fb560268a014101a9f7d16d308c9931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:19.301009 21576 raft_consensus.cc:1261] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 1 FOLLOWER]: Refusing update from remote peer 0fb560268a014101a9f7d16d308c9931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:19.301999 21650 consensus_queue.cc:1035] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad6ead3587f4372b2a308ddf9445a05" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:19.302342 21646 consensus_queue.cc:1035] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca30cd7559d6419291364a62318b29db" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:19.327378 21164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:19.327653 21164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:19.330953 21164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30ffd12654ee4adda9feed0ec56b2258: Sending DeleteTablet for 3 replicas of tablet 921298eb1b0649be829fedfef306bf5d
I20260924 11:52:19.332105 21556 tablet_service.cc:1514] Processing DeleteTablet for tablet 921298eb1b0649be829fedfef306bf5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:19 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:58214
I20260924 11:52:19.332244 21422 tablet_service.cc:1514] Processing DeleteTablet for tablet 921298eb1b0649be829fedfef306bf5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:19 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:57012
I20260924 11:52:19.332765 21288 tablet_service.cc:1514] Processing DeleteTablet for tablet 921298eb1b0649be829fedfef306bf5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:19 UTC) from {username='kudu', principal='kudu/127.1.210.62@KRBTEST.COM'} at 127.0.0.1:53596
I20260924 11:52:19.333241 21663 tablet_replica.cc:330] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: stopping tablet replica
I20260924 11:52:19.333849 21663 raft_consensus.cc:2229] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:19.334501 21663 raft_consensus.cc:2258] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:19.334604 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21225
I20260924 11:52:19.335421 21664 tablet_replica.cc:330] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: stopping tablet replica
I20260924 11:52:19.335860 21664 raft_consensus.cc:2229] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:52:19.336572 21664 raft_consensus.cc:2258] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:19.336968 21663 ts_tablet_manager.cc:1905] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:19.338697 21664 ts_tablet_manager.cc:1905] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:19.344152 21663 ts_tablet_manager.cc:1918] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:19.344337 21663 log.cc:1196] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/921298eb1b0649be829fedfef306bf5d
I20260924 11:52:19.344846 21663 ts_tablet_manager.cc:1939] T 921298eb1b0649be829fedfef306bf5d P ca30cd7559d6419291364a62318b29db: Deleting consensus metadata
I20260924 11:52:19.346128 21664 ts_tablet_manager.cc:1918] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:19.346338 21664 log.cc:1196] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/921298eb1b0649be829fedfef306bf5d
I20260924 11:52:19.346581 21150 catalog_manager.cc:4841] TS ca30cd7559d6419291364a62318b29db (127.1.210.3:40573): tablet 921298eb1b0649be829fedfef306bf5d (table test-table [id=7072133d8e5941f0b72c740fe4308b74]) successfully deleted
I20260924 11:52:19.347023 21664 ts_tablet_manager.cc:1939] T 921298eb1b0649be829fedfef306bf5d P 0fb560268a014101a9f7d16d308c9931: Deleting consensus metadata
I20260924 11:52:19.348310 21150 catalog_manager.cc:4841] TS 0fb560268a014101a9f7d16d308c9931 (127.1.210.2:46409): tablet 921298eb1b0649be829fedfef306bf5d (table test-table [id=7072133d8e5941f0b72c740fe4308b74]) successfully deleted
I20260924 11:52:19.349668 21665 tablet_replica.cc:330] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: stopping tablet replica
I20260924 11:52:19.350029 21665 raft_consensus.cc:2229] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:19.350440 21665 raft_consensus.cc:2258] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:19.352255 21665 ts_tablet_manager.cc:1905] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:19.357951 21665 ts_tablet_manager.cc:1918] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:19.358124 21665 log.cc:1196] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/921298eb1b0649be829fedfef306bf5d
I20260924 11:52:19.358698 21665 ts_tablet_manager.cc:1939] T 921298eb1b0649be829fedfef306bf5d P 2ad6ead3587f4372b2a308ddf9445a05: Deleting consensus metadata
Sep 24 11:52:19 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.1@KRBTEST.COM
I20260924 11:52:19.360353 21151 catalog_manager.cc:4841] TS 2ad6ead3587f4372b2a308ddf9445a05 (127.1.210.1:37421): tablet 921298eb1b0649be829fedfef306bf5d (table test-table [id=7072133d8e5941f0b72c740fe4308b74]) successfully deleted
I20260924 11:52:19.366256 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.335896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.1.210.1:37421)
0924 11:52:19.336073 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.336079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.348902 (+ 12823us) server_negotiation.cc:407] Connection header received
0924 11:52:19.349138 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.349146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.349295 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.349411 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:19.351098 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.352288 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.354326 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.354957 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.355089 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.360532 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:19.360546 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:19.360562 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:19.360610 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:19.363508 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:19.364321 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:19.364327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:19.364329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:19.364454 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:19.364829 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:19.364832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:19.364834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:19.365243 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:19.365465 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.365809 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.365961 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:52:19.366888 21348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:19.513346 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21225
I20260924 11:52:19.537247 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21359
Sep 24 11:52:19 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.2@KRBTEST.COM
I20260924 11:52:19.555799 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.538152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.1.210.2:46409)
0924 11:52:19.538343 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.538348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.538444 (+ 96us) server_negotiation.cc:407] Connection header received
0924 11:52:19.538731 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.538736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.538831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.538910 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:19.540856 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.541798 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.543794 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.544271 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.544366 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.549817 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:19.549831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:19.549846 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:19.549882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:19.552881 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:19.553619 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:19.553627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:19.553629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:19.553817 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:19.554191 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:19.554198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:19.554203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:19.554628 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:19.554961 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.555183 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.555410 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:52:19.556453 21482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:19.689841 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21359
I20260924 11:52:19.709017 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21493
Sep 24 11:52:19 dist-test-slave-dd32 krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.3@KRBTEST.COM
I20260924 11:52:19.723682 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:19.709787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.1.210.3:40573)
0924 11:52:19.709951 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:19.709956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:19.710015 (+ 59us) server_negotiation.cc:407] Connection header received
0924 11:52:19.710161 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:19.710166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:19.710239 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:19.710322 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:19.711539 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.712243 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.713536 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:19.713907 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:19.713998 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:19.718493 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:19.718507 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:19.718523 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:19.718563 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:19.721212 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:19.721909 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:19.721918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:19.721920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:19.722074 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:19.722415 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:19.722418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:19.722420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:19.722789 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:19.723017 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:19.723259 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 11:52:19.723429 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:52:19.724490 21616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:19.813257 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21493
I20260924 11:52:19.829789 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21132
I20260924 11:52:19.830852 21194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:19.942446 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21132
2026-09-24T11:52:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4550 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T11:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:20Z Disabled control of system clock
I20260924 11:52:20.019320 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43103
--webserver_interface=127.1.210.62
--webserver_port=0
--builtin_ntp_servers=127.1.210.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:52:20.342078 21674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:20.342396 21674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:20.342571 21674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:20.349757 21674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:20.349859 21674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:20.349959 21674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:20.350066 21674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:20.350193 21674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:20.359319 21674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:20.360641 21674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:20.362468 21674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:20.374322 21682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:20.374366 21680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:20.374842 21679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:20.375257 21674 server_base.cc:1034] running on GCE node
I20260924 11:52:20.376021 21674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:20.377460 21674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:20.378693 21674 hybrid_clock.cc:648] HybridClock initialized: now 1790250740378647 us; error 36 us; skew 500 ppm
I20260924 11:52:20.379035 21674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:20.384894 21674 webserver.cc:460] Webserver started at http://127.1.210.62:34459/ using document root <none> and password file <none>
I20260924 11:52:20.385627 21674 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:20.385768 21674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:20.386083 21674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:20.388893 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "043946c382bb476db3d6a60c76d0f9d4"
format_stamp: "Formatted at 2026-09-24 11:52:20 on dist-test-slave-dd32"
server_key: "e1ebd8190e84fb946857c19f50922eef"
server_key_iv: "238b9f6b848d988a371e3223e6983919"
server_key_version: "encryptionkey@0"
I20260924 11:52:20.389744 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "043946c382bb476db3d6a60c76d0f9d4"
format_stamp: "Formatted at 2026-09-24 11:52:20 on dist-test-slave-dd32"
server_key: "e1ebd8190e84fb946857c19f50922eef"
server_key_iv: "238b9f6b848d988a371e3223e6983919"
server_key_version: "encryptionkey@0"
I20260924 11:52:20.394707 21674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:52:20.397935 21688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:20.399142 21674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:20.399334 21674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "043946c382bb476db3d6a60c76d0f9d4"
format_stamp: "Formatted at 2026-09-24 11:52:20 on dist-test-slave-dd32"
server_key: "e1ebd8190e84fb946857c19f50922eef"
server_key_iv: "238b9f6b848d988a371e3223e6983919"
server_key_version: "encryptionkey@0"
I20260924 11:52:20.399542 21674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:20.411818 21674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:20.412706 21674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:20.413025 21674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:20.431412 21674 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43103
I20260924 11:52:20.431432 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43103 every 8 connection(s)
I20260924 11:52:20.432858 21674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:20.436668 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:20.442616 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21674
I20260924 11:52:20.442834 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:20.443272 1864 external_mini_cluster.cc:1428] Setting key cbc1f23324aed1be427debb57ab804c5
I20260924 11:52:20.448639 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4: Bootstrap starting.
I20260924 11:52:20.452656 21740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:20.453787 21740 log.cc:824] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:20.457196 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4: No bootstrap required, opened a new log
I20260924 11:52:20.458643 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:20.445381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.1.210.62:43103)
0924 11:52:20.446523 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:20.446538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:20.446621 (+ 83us) server_negotiation.cc:407] Connection header received
0924 11:52:20.448987 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:20.449016 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:20.449674 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:20.450094 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:20.450704 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:20.452616 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:20.454072 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:20.454953 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:20.455141 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:20.456232 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:20.456243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:20.456262 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:20.456960 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:20.457075 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:20.457103 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:20.457247 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:20.457464 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 11:52:20.457768 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":363,"threads_started":1}
I20260924 11:52:20.465444 21740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043946c382bb476db3d6a60c76d0f9d4" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43103 } }
I20260924 11:52:20.465803 21740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:20.465930 21740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043946c382bb476db3d6a60c76d0f9d4, State: Initialized, Role: FOLLOWER
I20260924 11:52:20.466580 21740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043946c382bb476db3d6a60c76d0f9d4" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43103 } }
I20260924 11:52:20.466869 21740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:20.467000 21740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:20.467177 21740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:20.469336 21740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043946c382bb476db3d6a60c76d0f9d4" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43103 } }
I20260924 11:52:20.469808 21740 leader_election.cc:302] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043946c382bb476db3d6a60c76d0f9d4; no voters:
I20260924 11:52:20.470407 21740 leader_election.cc:288] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:20.470516 21745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:20.471540 21745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [term 1 LEADER]: Becoming Leader. State: Replica: 043946c382bb476db3d6a60c76d0f9d4, State: Running, Role: LEADER
I20260924 11:52:20.472013 21745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043946c382bb476db3d6a60c76d0f9d4" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43103 } }
I20260924 11:52:20.472494 21740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:20.475035 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043946c382bb476db3d6a60c76d0f9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043946c382bb476db3d6a60c76d0f9d4" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43103 } } }
I20260924 11:52:20.475517 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:20.475034 21747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043946c382bb476db3d6a60c76d0f9d4. Latest consensus state: current_term: 1 leader_uuid: "043946c382bb476db3d6a60c76d0f9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043946c382bb476db3d6a60c76d0f9d4" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43103 } } }
I20260924 11:52:20.475764 21747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:20.477406 21751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:20.483322 21751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:20.491753 21751 catalog_manager.cc:1347] Generated new cluster ID: 7eb4d16b52bf4a559b04b02abecb49bd
I20260924 11:52:20.491856 21751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:20.502887 21751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:20.504750 21751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:20.524103 21751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4: Generated new TSK 0
I20260924 11:52:20.525024 21751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:20.531615 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.1:0
--local_ip_for_outbound_sockets=127.1.210.1
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43103
--builtin_ntp_servers=127.1.210.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:52:20.879541 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:20.879841 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:20.880028 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:20.887730 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:20.887840 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:20.888001 21764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.1
I20260924 11:52:20.897146 21764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.1
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.1
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:20.898510 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:20.900362 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:20.914304 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:20.914331 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:20.914191 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:20.915253 21764 server_base.cc:1034] running on GCE node
I20260924 11:52:20.915894 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:20.916826 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:20.918064 21764 hybrid_clock.cc:648] HybridClock initialized: now 1790250740917976 us; error 77 us; skew 500 ppm
I20260924 11:52:20.918391 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:20.920554 21764 webserver.cc:460] Webserver started at http://127.1.210.1:46273/ using document root <none> and password file <none>
I20260924 11:52:20.921293 21764 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:20.921422 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:20.921741 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:20.924502 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data/instance:
uuid: "0be5bb1e45a6408c99edc2a54ac8b00d"
format_stamp: "Formatted at 2026-09-24 11:52:20 on dist-test-slave-dd32"
server_key: "f6577912d2e4c5bde82a3ab7de2e1990"
server_key_iv: "d4f673d4be6db8b6bda0a688129fb8ed"
server_key_version: "encryptionkey@0"
I20260924 11:52:20.925344 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance:
uuid: "0be5bb1e45a6408c99edc2a54ac8b00d"
format_stamp: "Formatted at 2026-09-24 11:52:20 on dist-test-slave-dd32"
server_key: "f6577912d2e4c5bde82a3ab7de2e1990"
server_key_iv: "d4f673d4be6db8b6bda0a688129fb8ed"
server_key_version: "encryptionkey@0"
I20260924 11:52:20.932291 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260924 11:52:20.936193 21778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:20.937534 21764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:20.937703 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal
uuid: "0be5bb1e45a6408c99edc2a54ac8b00d"
format_stamp: "Formatted at 2026-09-24 11:52:20 on dist-test-slave-dd32"
server_key: "f6577912d2e4c5bde82a3ab7de2e1990"
server_key_iv: "d4f673d4be6db8b6bda0a688129fb8ed"
server_key_version: "encryptionkey@0"
I20260924 11:52:20.937940 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:20.952085 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:20.952903 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:20.953249 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:20.954268 21764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:20.955770 21764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:20.955855 21764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:20.956002 21764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:20.956096 21764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:20.988930 21764 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.1:46243
I20260924 11:52:20.988984 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.1:46243 every 8 connection(s)
I20260924 11:52:20.990255 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/data/info.pb
I20260924 11:52:21.000268 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21764
I20260924 11:52:21.000456 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal/instance
I20260924 11:52:21.000875 1864 external_mini_cluster.cc:1428] Setting key dc7d5338f8ceef97c200109df40433ba
I20260924 11:52:21.002498 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:20.992752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.1:56247 (local address 127.1.210.62:43103)
0924 11:52:20.993220 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:20.993225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:20.994277 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 11:52:20.996049 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:20.996055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:20.996154 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:20.996272 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:20.997348 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:20.998096 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:20.999704 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.000090 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.000225 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.001679 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:21.001685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:21.001688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:21.001705 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:21.001746 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:21.001760 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:21.001858 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:21.002074 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 11:52:21.002170 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260924 11:52:21.003521 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:20.993251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43103 (local address 127.1.210.1:56247)
0924 11:52:20.994099 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:20.994159 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:20.995790 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:20.996493 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:20.996501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:20.996580 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:20.997185 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:20.997202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:20.998248 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:20.998253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:20.999552 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:20.999566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.000309 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.000319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.000575 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.001337 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:21.001340 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:21.001376 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:21.001447 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:21.001857 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:21.001869 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:21.001879 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:52:21.002047 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 11:52:21.002418 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":248,"threads_started":1}
I20260924 11:52:21.003545 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.2:0
--local_ip_for_outbound_sockets=127.1.210.2
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43103
--builtin_ntp_servers=127.1.210.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:52:21.005055 21891 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43103
I20260924 11:52:21.005393 21891 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:21.006219 21891 heartbeater.cc:511] Master 127.1.210.62:43103 requested a full tablet report, sending...
I20260924 11:52:21.008256 21705 ts_manager.cc:194] Registered new tserver with Master: 0be5bb1e45a6408c99edc2a54ac8b00d (127.1.210.1:46243)
I20260924 11:52:21.009774 21705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.1:56247
W20260924 11:52:21.330399 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:21.330669 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:21.330816 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:21.338152 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:21.338269 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:21.338481 21895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.2
I20260924 11:52:21.347546 21895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.2
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.2
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:21.348953 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:21.350744 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:21.364290 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:21.364236 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:21.364319 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:21.364997 21895 server_base.cc:1034] running on GCE node
I20260924 11:52:21.365643 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:21.366581 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:21.367780 21895 hybrid_clock.cc:648] HybridClock initialized: now 1790250741367742 us; error 26 us; skew 500 ppm
I20260924 11:52:21.368100 21895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:21.370440 21895 webserver.cc:460] Webserver started at http://127.1.210.2:36781/ using document root <none> and password file <none>
I20260924 11:52:21.371125 21895 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:21.371245 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:21.371572 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:21.374660 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data/instance:
uuid: "93e4424c8094473c892fa31c6ca6268f"
format_stamp: "Formatted at 2026-09-24 11:52:21 on dist-test-slave-dd32"
server_key: "e5834d99407835b5439cd0c9ab76dcb0"
server_key_iv: "843bad5ba7605d81e26f6d8ac1291e74"
server_key_version: "encryptionkey@0"
I20260924 11:52:21.375440 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance:
uuid: "93e4424c8094473c892fa31c6ca6268f"
format_stamp: "Formatted at 2026-09-24 11:52:21 on dist-test-slave-dd32"
server_key: "e5834d99407835b5439cd0c9ab76dcb0"
server_key_iv: "843bad5ba7605d81e26f6d8ac1291e74"
server_key_version: "encryptionkey@0"
I20260924 11:52:21.380678 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:52:21.384197 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:21.385593 21895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:21.385774 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal
uuid: "93e4424c8094473c892fa31c6ca6268f"
format_stamp: "Formatted at 2026-09-24 11:52:21 on dist-test-slave-dd32"
server_key: "e5834d99407835b5439cd0c9ab76dcb0"
server_key_iv: "843bad5ba7605d81e26f6d8ac1291e74"
server_key_version: "encryptionkey@0"
I20260924 11:52:21.385998 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:21.401907 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:21.402848 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:21.403167 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:21.404440 21895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:21.405992 21895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:21.406116 21895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:21.406262 21895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:21.406353 21895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:21.441924 21895 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.2:37357
I20260924 11:52:21.441948 22021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.2:37357 every 8 connection(s)
I20260924 11:52:21.443200 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/data/info.pb
I20260924 11:52:21.450078 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 21895
I20260924 11:52:21.450249 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal/instance
I20260924 11:52:21.450695 1864 external_mini_cluster.cc:1428] Setting key cfa967b36a521f9f69b6fae3815cf69a
I20260924 11:52:21.453377 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.3:0
--local_ip_for_outbound_sockets=127.1.210.3
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43103
--builtin_ntp_servers=127.1.210.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:52:21.456141 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.445921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.2:47963 (local address 127.1.210.62:43103)
0924 11:52:21.446130 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:21.446136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:21.447604 (+ 1468us) server_negotiation.cc:407] Connection header received
0924 11:52:21.449563 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:21.449567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:21.449632 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:21.449753 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:21.450863 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.451840 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.453410 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.453914 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.454048 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.455369 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:21.455375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:21.455379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:21.455406 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:21.455436 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:21.455447 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:21.455538 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:21.455756 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 11:52:21.455850 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:52:21.457204 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.446442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43103 (local address 127.1.210.2:47963)
0924 11:52:21.447445 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:21.447487 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:21.449336 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:21.449929 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:21.449937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:21.450019 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:21.450670 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.450686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.451983 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.451990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.453270 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.453281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.454138 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.454145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.454375 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.455065 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:21.455067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:21.455105 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:21.455176 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:21.455536 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:21.455548 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:21.455558 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:52:21.455742 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 11:52:21.456123 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260924 11:52:21.458739 22022 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43103
I20260924 11:52:21.459045 22022 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:21.459771 22022 heartbeater.cc:511] Master 127.1.210.62:43103 requested a full tablet report, sending...
I20260924 11:52:21.461186 21705 ts_manager.cc:194] Registered new tserver with Master: 93e4424c8094473c892fa31c6ca6268f (127.1.210.2:37357)
I20260924 11:52:21.461988 21705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.2:47963
W20260924 11:52:21.782291 22025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:21.782608 22025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:21.782764 22025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:21.790733 22025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:21.790877 22025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:21.791110 22025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.3
I20260924 11:52:21.803570 22025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.3
--webserver_port=0
--tserver_master_addrs=127.1.210.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.3
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:21.804859 22025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:21.806697 22025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:21.819402 22033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:21.819489 22030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:21.819478 22031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:21.822331 22025 server_base.cc:1034] running on GCE node
I20260924 11:52:21.822919 22025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:21.823812 22025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:21.825006 22025 hybrid_clock.cc:648] HybridClock initialized: now 1790250741824969 us; error 34 us; skew 500 ppm
I20260924 11:52:21.825346 22025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:52:21.827687 22025 webserver.cc:460] Webserver started at http://127.1.210.3:33363/ using document root <none> and password file <none>
I20260924 11:52:21.828397 22025 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:21.828514 22025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:21.828846 22025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:21.831869 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data/instance:
uuid: "b3b620ff97fe4cb1b50add36cd06a3b3"
format_stamp: "Formatted at 2026-09-24 11:52:21 on dist-test-slave-dd32"
server_key: "cd602e107377dcbae33a55b12945e0d0"
server_key_iv: "ee416b69228629f761b0ed96e050ccc1"
server_key_version: "encryptionkey@0"
I20260924 11:52:21.832659 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance:
uuid: "b3b620ff97fe4cb1b50add36cd06a3b3"
format_stamp: "Formatted at 2026-09-24 11:52:21 on dist-test-slave-dd32"
server_key: "cd602e107377dcbae33a55b12945e0d0"
server_key_iv: "ee416b69228629f761b0ed96e050ccc1"
server_key_version: "encryptionkey@0"
I20260924 11:52:21.838163 22025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:52:21.841868 22039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:21.843401 22025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:21.843585 22025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal
uuid: "b3b620ff97fe4cb1b50add36cd06a3b3"
format_stamp: "Formatted at 2026-09-24 11:52:21 on dist-test-slave-dd32"
server_key: "cd602e107377dcbae33a55b12945e0d0"
server_key_iv: "ee416b69228629f761b0ed96e050ccc1"
server_key_version: "encryptionkey@0"
I20260924 11:52:21.843809 22025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:21.870678 22025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:21.871541 22025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:21.871831 22025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:21.872884 22025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:52:21.874394 22025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:52:21.874506 22025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:21.874640 22025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:52:21.874728 22025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:21.908406 22025 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.3:39561
I20260924 11:52:21.908500 22151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.3:39561 every 8 connection(s)
I20260924 11:52:21.909839 22025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/data/info.pb
I20260924 11:52:21.918499 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22025
I20260924 11:52:21.918696 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal/instance
I20260924 11:52:21.919111 1864 external_mini_cluster.cc:1428] Setting key e74a043a595df690c9107f9b036fcafa
I20260924 11:52:21.922653 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.912704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:45655 (local address 127.1.210.62:43103)
0924 11:52:21.912887 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:21.912892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:21.914392 (+ 1500us) server_negotiation.cc:407] Connection header received
0924 11:52:21.916140 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:21.916144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:21.916209 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:21.916290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:21.917405 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.918151 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.919887 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.920345 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.920476 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.921853 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:21.921858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:21.921861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:21.921878 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:21.921909 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:21.921920 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:21.922017 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:21.922251 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 11:52:21.922347 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:52:21.923650 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.913221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:43103 (local address 127.1.210.3:45655)
0924 11:52:21.914238 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:21.914287 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:21.915887 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:21.916490 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:21.916498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:21.916583 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:21.917257 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.917280 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.918326 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.918331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.919704 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.919745 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.920572 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.920579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.920800 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.921497 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:52:21.921500 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:21.921541 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:52:21.921615 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:21.922016 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:21.922029 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:21.922039 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:52:21.922240 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 11:52:21.922625 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":324,"threads_started":1}
I20260924 11:52:21.924911 22152 heartbeater.cc:347] Connected to a master server at 127.1.210.62:43103
I20260924 11:52:21.925252 22152 heartbeater.cc:464] Registering TS with master...
I20260924 11:52:21.925942 22152 heartbeater.cc:511] Master 127.1.210.62:43103 requested a full tablet report, sending...
I20260924 11:52:21.927278 21705 ts_manager.cc:194] Registered new tserver with Master: b3b620ff97fe4cb1b50add36cd06a3b3 (127.1.210.3:39561)
I20260924 11:52:21.928153 21705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.3:45655
I20260924 11:52:21.934129 1864 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:52:21.943877 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.938317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.1.210.62:43103)
0924 11:52:21.938482 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:21.938486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:21.938931 (+ 445us) server_negotiation.cc:407] Connection header received
0924 11:52:21.939065 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:21.939069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:21.939138 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:21.939221 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:21.939684 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.940391 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.941809 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.942261 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.942354 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.943342 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:21.943347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:21.943350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:21.943366 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:21.943396 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:21.943407 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:21.943474 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:21.943588 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:52:21.943658 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:52:21.947785 21705 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:52:21.954681 21705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:52:21.993119 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.967843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.1.210.2:37357)
0924 11:52:21.979810 (+ 11967us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:21.979825 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:21.981401 (+ 1576us) server_negotiation.cc:407] Connection header received
0924 11:52:21.981668 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:21.981677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:21.981940 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:21.982156 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:21.983088 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.984980 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.988564 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.990218 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.990384 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.990721 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:21.992398 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0924 11:52:21.992648 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11805,"thread_start_us":11695,"threads_started":1}
I20260924 11:52:21.993574 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.967643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:37357 (local address 127.0.0.1:35706)
0924 11:52:21.981271 (+ 13628us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:21.981290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:21.981438 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:21.982449 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:21.982456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:21.982479 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:21.982895 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.982909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.985209 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.985216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.988325 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.988347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.990889 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.990898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.991202 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:21.991272 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 11:52:21.991435 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 11:52:21.991695 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13524,"thread_start_us":11010,"threads_started":1}
I20260924 11:52:21.999171 21957 tablet_service.cc:1467] Processing CreateTablet for tablet 0b79bddcf2e741dbac31abeaa412f8c0 (DEFAULT_TABLE table=test-table [id=22799b620fdb4f24a611f2d580055b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:22.000931 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b79bddcf2e741dbac31abeaa412f8c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:22.002446 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.967176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.3:39561 (local address 127.0.0.1:46358)
0924 11:52:21.978178 (+ 11002us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:21.978229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:21.978422 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:21.980767 (+ 2345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:21.980778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:21.980826 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:21.981438 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.981454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.984011 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.984020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:21.987406 (+ 3386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:21.987428 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.001221 (+ 13793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.001232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.001536 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.001596 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 11:52:22.001683 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:52:22.001910 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10679,"mutex_wait_us":1,"thread_start_us":13797,"threads_started":1}
I20260924 11:52:22.004027 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.967202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.1.210.3:39561)
0924 11:52:21.980208 (+ 13006us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:21.980215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:21.980243 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:52:21.980320 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:21.980325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:21.980519 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:21.980666 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:21.981689 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:21.983807 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:21.987790 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.000919 (+ 13129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.002743 (+ 1824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.003140 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.003256 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.003514 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12874,"thread_start_us":12751,"threads_started":1}
I20260924 11:52:22.009845 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.981661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46243 (local address 127.0.0.1:33130)
0924 11:52:21.995314 (+ 13653us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:21.995341 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:21.995471 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:22.002895 (+ 7424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:22.002900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:22.002928 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:22.003119 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:22.003131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.005617 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.005622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.007543 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:22.007559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.009027 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.009033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.009255 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.009301 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 11:52:22.009403 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 11:52:22.009544 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13507,"thread_start_us":26140,"threads_started":1}
I20260924 11:52:22.009977 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:21.981717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.1.210.1:46243)
0924 11:52:22.000458 (+ 18741us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.000468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.000501 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:52:22.000613 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.000620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.000855 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.001057 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:22.003321 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.005419 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.007756 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.008843 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.009012 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.009312 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.009452 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.009612 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18553,"thread_start_us":19336,"threads_started":1}
I20260924 11:52:22.011969 22087 tablet_service.cc:1467] Processing CreateTablet for tablet 0b79bddcf2e741dbac31abeaa412f8c0 (DEFAULT_TABLE table=test-table [id=22799b620fdb4f24a611f2d580055b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:22.013571 22087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b79bddcf2e741dbac31abeaa412f8c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:22.013073 21891 heartbeater.cc:503] Master 127.1.210.62:43103 was elected leader, sending a full tablet report...
I20260924 11:52:22.013299 21826 tablet_service.cc:1467] Processing CreateTablet for tablet 0b79bddcf2e741dbac31abeaa412f8c0 (DEFAULT_TABLE table=test-table [id=22799b620fdb4f24a611f2d580055b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:52:22.015995 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b79bddcf2e741dbac31abeaa412f8c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:22.017297 22170 tablet_bootstrap.cc:492] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Bootstrap starting.
I20260924 11:52:22.020871 22170 tablet_bootstrap.cc:654] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:22.022187 22170 log.cc:824] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:22.025386 22170 tablet_bootstrap.cc:492] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: No bootstrap required, opened a new log
I20260924 11:52:22.025650 22170 ts_tablet_manager.cc:1397] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 11:52:22.026072 22172 tablet_bootstrap.cc:492] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Bootstrap starting.
I20260924 11:52:22.029224 22172 tablet_bootstrap.cc:654] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:22.030375 22173 tablet_bootstrap.cc:492] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Bootstrap starting.
I20260924 11:52:22.030597 22172 log.cc:824] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:22.033684 22173 tablet_bootstrap.cc:654] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:22.033748 22172 tablet_bootstrap.cc:492] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: No bootstrap required, opened a new log
I20260924 11:52:22.033987 22172 ts_tablet_manager.cc:1397] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 11:52:22.034865 22173 log.cc:824] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:22.035104 22170 raft_consensus.cc:348] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.035595 22170 raft_consensus.cc:374] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:22.035753 22170 raft_consensus.cc:729] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e4424c8094473c892fa31c6ca6268f, State: Initialized, Role: FOLLOWER
I20260924 11:52:22.036396 22170 consensus_queue.cc:260] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.037487 22022 heartbeater.cc:503] Master 127.1.210.62:43103 was elected leader, sending a full tablet report...
I20260924 11:52:22.037935 22173 tablet_bootstrap.cc:492] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: No bootstrap required, opened a new log
I20260924 11:52:22.038137 22170 ts_tablet_manager.cc:1428] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 11:52:22.038172 22173 ts_tablet_manager.cc:1397] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:52:22.044687 22172 raft_consensus.cc:348] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.045200 22172 raft_consensus.cc:374] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:22.045302 22172 raft_consensus.cc:729] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b620ff97fe4cb1b50add36cd06a3b3, State: Initialized, Role: FOLLOWER
I20260924 11:52:22.045924 22172 consensus_queue.cc:260] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.046998 22152 heartbeater.cc:503] Master 127.1.210.62:43103 was elected leader, sending a full tablet report...
I20260924 11:52:22.047808 22172 ts_tablet_manager.cc:1428] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 11:52:22.048607 22173 raft_consensus.cc:348] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.049031 22173 raft_consensus.cc:374] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:22.049216 22173 raft_consensus.cc:729] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be5bb1e45a6408c99edc2a54ac8b00d, State: Initialized, Role: FOLLOWER
I20260924 11:52:22.049831 22173 consensus_queue.cc:260] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.051672 22173 ts_tablet_manager.cc:1428] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 11:52:22.123373 22177 raft_consensus.cc:482] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:52:22.123764 22177 raft_consensus.cc:504] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.125643 22177 leader_election.cc:288] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be5bb1e45a6408c99edc2a54ac8b00d (127.1.210.1:46243), 93e4424c8094473c892fa31c6ca6268f (127.1.210.2:37357)
I20260924 11:52:22.132292 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.126274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:48533 (local address 127.1.210.2:37357)
0924 11:52:22.126501 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.126509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.126632 (+ 123us) server_negotiation.cc:407] Connection header received
0924 11:52:22.126924 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.126930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.127030 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.127148 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:22.127646 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.128643 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.130476 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.131382 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.131476 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.131540 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.131901 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.132030 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:52:22.132283 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.126088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.2:37357 (local address 127.1.210.3:48533)
0924 11:52:22.126504 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:22.126533 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:22.126754 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:22.127235 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:22.127262 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:22.127291 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:22.127480 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:22.127496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.128788 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.128793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.130317 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:22.130328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.131535 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.131542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.131714 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.131754 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:52:22.131861 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 11:52:22.132031 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260924 11:52:22.132732 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.126128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.3:54555 (local address 127.1.210.1:46243)
0924 11:52:22.126374 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.126381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.127143 (+ 762us) server_negotiation.cc:407] Connection header received
0924 11:52:22.127326 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.127333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.127419 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.127521 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:52:22.128221 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.129088 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.130910 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.131833 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.131946 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.132045 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.132387 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.132528 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:52:22.133098 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.126165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.1:46243 (local address 127.1.210.3:54555)
0924 11:52:22.127020 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:22.127039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:22.127179 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:22.127546 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:22.127558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:22.127606 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:52:22.128012 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:22.128025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.129232 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.129235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.130724 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:22.130737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.131997 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.132002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:22.132185 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.132224 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:52:22.132333 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 11:52:22.132772 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":1,"thread_start_us":482,"threads_started":1}
I20260924 11:52:22.133466 21977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b79bddcf2e741dbac31abeaa412f8c0" candidate_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e4424c8094473c892fa31c6ca6268f" is_pre_election: true
I20260924 11:52:22.133939 21977 raft_consensus.cc:2454] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b620ff97fe4cb1b50add36cd06a3b3 in term 0.
I20260924 11:52:22.133994 21846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b79bddcf2e741dbac31abeaa412f8c0" candidate_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" is_pre_election: true
I20260924 11:52:22.134526 21846 raft_consensus.cc:2454] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b620ff97fe4cb1b50add36cd06a3b3 in term 0.
I20260924 11:52:22.134584 22042 leader_election.cc:302] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93e4424c8094473c892fa31c6ca6268f, b3b620ff97fe4cb1b50add36cd06a3b3; no voters:
I20260924 11:52:22.134997 22177 raft_consensus.cc:2781] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:52:22.135133 22177 raft_consensus.cc:482] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:52:22.135258 22177 raft_consensus.cc:3037] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:22.137565 22177 raft_consensus.cc:504] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.138159 22177 leader_election.cc:288] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0be5bb1e45a6408c99edc2a54ac8b00d (127.1.210.1:46243), 93e4424c8094473c892fa31c6ca6268f (127.1.210.2:37357)
I20260924 11:52:22.138559 21846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b79bddcf2e741dbac31abeaa412f8c0" candidate_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d"
I20260924 11:52:22.138628 21977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b79bddcf2e741dbac31abeaa412f8c0" candidate_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e4424c8094473c892fa31c6ca6268f"
I20260924 11:52:22.138787 21846 raft_consensus.cc:3037] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:22.138871 21977 raft_consensus.cc:3037] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:22.141022 21846 raft_consensus.cc:2454] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b620ff97fe4cb1b50add36cd06a3b3 in term 1.
I20260924 11:52:22.141109 21977 raft_consensus.cc:2454] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b620ff97fe4cb1b50add36cd06a3b3 in term 1.
I20260924 11:52:22.141474 22041 leader_election.cc:302] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be5bb1e45a6408c99edc2a54ac8b00d, b3b620ff97fe4cb1b50add36cd06a3b3; no voters:
I20260924 11:52:22.141839 22177 raft_consensus.cc:2781] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:22.142318 22177 raft_consensus.cc:686] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 1 LEADER]: Becoming Leader. State: Replica: b3b620ff97fe4cb1b50add36cd06a3b3, State: Running, Role: LEADER
I20260924 11:52:22.142805 22177 consensus_queue.cc:237] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } }
I20260924 11:52:22.147236 21705 catalog_manager.cc:5495] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b620ff97fe4cb1b50add36cd06a3b3 (127.1.210.3). New cstate: current_term: 1 leader_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b620ff97fe4cb1b50add36cd06a3b3" member_type: VOTER last_known_addr { host: "127.1.210.3" port: 39561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 } health_report { overall_health: UNKNOWN } } }
W20260924 11:52:22.162781 22153 tablet.cc:2369] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:52:22.167759 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.162480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.1.210.3:39561)
0924 11:52:22.162717 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.162724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.162746 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:52:22.162850 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.162855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.162970 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.163128 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:52:22.163519 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.164309 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.165784 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.166171 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.166267 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.166684 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:52:22.167183 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:52:22.167304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.167399 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.167463 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:52:22.176661 21977 raft_consensus.cc:1261] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 1 FOLLOWER]: Refusing update from remote peer b3b620ff97fe4cb1b50add36cd06a3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:22.176694 21846 raft_consensus.cc:1261] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 1 FOLLOWER]: Refusing update from remote peer b3b620ff97fe4cb1b50add36cd06a3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:52:22.177836 22180 consensus_queue.cc:1035] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be5bb1e45a6408c99edc2a54ac8b00d" member_type: VOTER last_known_addr { host: "127.1.210.1" port: 46243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:22.178231 22177 consensus_queue.cc:1035] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e4424c8094473c892fa31c6ca6268f" member_type: VOTER last_known_addr { host: "127.1.210.2" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:52:22.201944 21705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:22.202216 21705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:52:22.206785 21705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043946c382bb476db3d6a60c76d0f9d4: Sending DeleteTablet for 3 replicas of tablet 0b79bddcf2e741dbac31abeaa412f8c0
I20260924 11:52:22.208065 22087 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b79bddcf2e741dbac31abeaa412f8c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:22 UTC) from {username='slave'} at 127.0.0.1:46358
I20260924 11:52:22.208345 21826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b79bddcf2e741dbac31abeaa412f8c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:22 UTC) from {username='slave'} at 127.0.0.1:33130
I20260924 11:52:22.209208 22194 tablet_replica.cc:330] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: stopping tablet replica
I20260924 11:52:22.209573 21957 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b79bddcf2e741dbac31abeaa412f8c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:52:22 UTC) from {username='slave'} at 127.0.0.1:35706
I20260924 11:52:22.209690 22194 raft_consensus.cc:2229] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:22.210281 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21764
I20260924 11:52:22.210338 22194 raft_consensus.cc:2258] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:22.213140 22194 ts_tablet_manager.cc:1905] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:22.225551 22193 tablet_replica.cc:330] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: stopping tablet replica
I20260924 11:52:22.225832 22195 tablet_replica.cc:330] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: stopping tablet replica
I20260924 11:52:22.226014 22193 raft_consensus.cc:2229] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:52:22.226259 22195 raft_consensus.cc:2229] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:52:22.226297 22194 ts_tablet_manager.cc:1918] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:22.226462 22194 log.cc:1196] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-0/wal/wals/0b79bddcf2e741dbac31abeaa412f8c0
I20260924 11:52:22.226675 22193 raft_consensus.cc:2258] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:22.226804 22195 raft_consensus.cc:2258] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:52:22.226992 22194 ts_tablet_manager.cc:1939] T 0b79bddcf2e741dbac31abeaa412f8c0 P 0be5bb1e45a6408c99edc2a54ac8b00d: Deleting consensus metadata
I20260924 11:52:22.228860 22193 ts_tablet_manager.cc:1905] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:22.228857 21690 catalog_manager.cc:4841] TS 0be5bb1e45a6408c99edc2a54ac8b00d (127.1.210.1:46243): tablet 0b79bddcf2e741dbac31abeaa412f8c0 (table test-table [id=22799b620fdb4f24a611f2d580055b59]) successfully deleted
I20260924 11:52:22.230686 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.211179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.1.210.1:46243)
0924 11:52:22.211373 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.211382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.217493 (+ 6111us) server_negotiation.cc:407] Connection header received
0924 11:52:22.217687 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.217691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.217790 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.217884 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:22.221623 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.222810 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.224727 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.225414 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.225514 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.228990 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:22.229001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:22.229018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:22.229783 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:22.229849 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:22.229876 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:22.229956 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.230214 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.230335 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:52:22.231400 21886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:22.233933 22195 ts_tablet_manager.cc:1905] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:52:22.240849 22193 ts_tablet_manager.cc:1918] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:22.241043 22193 log.cc:1196] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-2/wal/wals/0b79bddcf2e741dbac31abeaa412f8c0
I20260924 11:52:22.241020 22195 ts_tablet_manager.cc:1918] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:52:22.241285 22195 log.cc:1196] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Deleting WAL directory at /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790250429340952-1864-0/minicluster-data/ts-1/wal/wals/0b79bddcf2e741dbac31abeaa412f8c0
I20260924 11:52:22.241631 22193 ts_tablet_manager.cc:1939] T 0b79bddcf2e741dbac31abeaa412f8c0 P b3b620ff97fe4cb1b50add36cd06a3b3: Deleting consensus metadata
I20260924 11:52:22.241851 22195 ts_tablet_manager.cc:1939] T 0b79bddcf2e741dbac31abeaa412f8c0 P 93e4424c8094473c892fa31c6ca6268f: Deleting consensus metadata
I20260924 11:52:22.243522 21690 catalog_manager.cc:4841] TS b3b620ff97fe4cb1b50add36cd06a3b3 (127.1.210.3:39561): tablet 0b79bddcf2e741dbac31abeaa412f8c0 (table test-table [id=22799b620fdb4f24a611f2d580055b59]) successfully deleted
I20260924 11:52:22.243697 21691 catalog_manager.cc:4841] TS 93e4424c8094473c892fa31c6ca6268f (127.1.210.2:37357): tablet 0b79bddcf2e741dbac31abeaa412f8c0 (table test-table [id=22799b620fdb4f24a611f2d580055b59]) successfully deleted
I20260924 11:52:22.324409 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21764
I20260924 11:52:22.341847 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21895
I20260924 11:52:22.349608 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.342674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.1.210.2:37357)
0924 11:52:22.342885 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.342892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.342914 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:52:22.342986 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.342992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.343112 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.343237 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:22.343577 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.344731 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.346014 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.346674 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.346791 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.347909 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:22.347920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:22.347938 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:22.348754 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:22.348824 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:22.348851 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:22.348958 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.349124 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.349243 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:52:22.350366 22017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:22.441262 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21895
I20260924 11:52:22.463745 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22025
I20260924 11:52:22.471577 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:22.464591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.1.210.3:39561)
0924 11:52:22.464818 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:22.464823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:22.464956 (+ 133us) server_negotiation.cc:407] Connection header received
0924 11:52:22.465175 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:22.465179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:22.465260 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:22.465344 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:22.465807 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.466607 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.468333 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:22.468767 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:22.468858 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:22.470122 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:22.470136 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:22.470150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:22.470792 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:22.470838 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:22.470857 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:22.470945 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:22.471113 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 11:52:22.471189 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:52:22.472358 22147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:22.593535 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22025
I20260924 11:52:22.613312 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 21674
I20260924 11:52:22.614380 21735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:22.722774 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 21674
2026-09-24T11:52:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2778 ms)
[----------] 6 tests from EncryptionPolicyTest (22057 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:22 dist-test-slave-dd32 krb5kdc[22203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:22 dist-test-slave-dd32 krb5kdc[22203](info): set up 2 sockets
Sep 24 11:52:22 dist-test-slave-dd32 krb5kdc[22203](info): commencing operation
krb5kdc: starting...
W20260924 11:52:23.833467 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:23 dist-test-slave-dd32 krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:24.085696 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:24.425297 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:24.425652 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:24.425829 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:24.433111 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:24.433234 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:24.433358 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:24.433476 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:24.433605 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:24.443991 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:24.445420 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:24.446811 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:24.458972 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:24.459085 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:24.459080 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:24.459837 22216 server_base.cc:1034] running on GCE node
I20260924 11:52:24.460664 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:24.462141 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:24.463393 22216 hybrid_clock.cc:648] HybridClock initialized: now 1790250744463357 us; error 51 us; skew 500 ppm
I20260924 11:52:24.463778 22216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:24 dist-test-slave-dd32 krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:24.467134 22216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:24.468614 22216 webserver.cc:460] Webserver started at http://127.0.0.1:34291/ using document root <none> and password file <none>
I20260924 11:52:24.469385 22216 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:24.469527 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:24.469832 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:24.472676 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "ffc2901863bb4bcd80d5df3e2554d3f9"
format_stamp: "Formatted at 2026-09-24 11:52:24 on dist-test-slave-dd32"
server_key: "ad5b038f9c16d935fb013c3da6d70eac"
server_key_iv: "58ed8b7f19dac4b1cf743efeb207bbc8"
server_key_version: "encryptionkey@0"
I20260924 11:52:24.473479 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "ffc2901863bb4bcd80d5df3e2554d3f9"
format_stamp: "Formatted at 2026-09-24 11:52:24 on dist-test-slave-dd32"
server_key: "ad5b038f9c16d935fb013c3da6d70eac"
server_key_iv: "58ed8b7f19dac4b1cf743efeb207bbc8"
server_key_version: "encryptionkey@0"
I20260924 11:52:24.478338 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:52:24.481802 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:24.483064 22216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:24.483242 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "ffc2901863bb4bcd80d5df3e2554d3f9"
format_stamp: "Formatted at 2026-09-24 11:52:24 on dist-test-slave-dd32"
server_key: "ad5b038f9c16d935fb013c3da6d70eac"
server_key_iv: "58ed8b7f19dac4b1cf743efeb207bbc8"
server_key_version: "encryptionkey@0"
I20260924 11:52:24.483428 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:24.498631 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:24.502473 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:24.502763 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:24.521183 22216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45781
I20260924 11:52:24.521186 22282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45781 every 8 connection(s)
I20260924 11:52:24.522567 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:24.526590 22283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:24.530166 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22216
I20260924 11:52:24.530398 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:24.531342 1864 external_mini_cluster.cc:1428] Setting key 877129a5b63cf31fd12b16178cfd2486
I20260924 11:52:24.539790 22283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9: Bootstrap starting.
I20260924 11:52:24.543277 22283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:24.544363 22283 log.cc:824] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:52:24 dist-test-slave-dd32 krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:52:24.547691 22283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9: No bootstrap required, opened a new log
I20260924 11:52:24.554363 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:24.533515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49909 (local address 127.0.0.1:45781)
0924 11:52:24.534306 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:24.534318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:24.534359 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:52:24.536095 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:24.536117 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:24.536542 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:24.536928 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:24.538408 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:24.539756 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:24.541027 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:24.541763 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:24.541954 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:24.546636 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:24.546644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:24.546658 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:24.546693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:24.549922 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:24.550950 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:24.550956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:24.550963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:24.551127 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:24.551623 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:24.551626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:24.551628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:24.552147 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:24.552385 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:24.552921 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 11:52:24.553328 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
I20260924 11:52:24.557278 22283 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45781 } }
I20260924 11:52:24.557752 22283 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:24.557895 22283 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc2901863bb4bcd80d5df3e2554d3f9, State: Initialized, Role: FOLLOWER
I20260924 11:52:24.558593 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:24.558569 22283 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45781 } }
I20260924 11:52:24.558872 22283 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:24.559029 22283 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:24.559198 22283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:24.562789 22283 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45781 } }
I20260924 11:52:24.563515 22283 leader_election.cc:302] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc2901863bb4bcd80d5df3e2554d3f9; no voters:
I20260924 11:52:24.564221 22283 leader_election.cc:288] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:24.564378 22292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:24.567279 22283 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:24.569484 22292 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [term 1 LEADER]: Becoming Leader. State: Replica: ffc2901863bb4bcd80d5df3e2554d3f9, State: Running, Role: LEADER
I20260924 11:52:24.570343 22292 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45781 } }
I20260924 11:52:24.579020 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc2901863bb4bcd80d5df3e2554d3f9. Latest consensus state: current_term: 1 leader_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45781 } } }
I20260924 11:52:24.579054 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc2901863bb4bcd80d5df3e2554d3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45781 } } }
I20260924 11:52:24.579438 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:24.579464 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:24.587380 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:24.564892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56019 (local address 127.0.0.1:45781)
0924 11:52:24.565313 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:24.565328 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:24.565880 (+ 552us) server_negotiation.cc:407] Connection header received
0924 11:52:24.566175 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:24.566181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:24.566273 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:24.566357 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:24.568806 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:24.570662 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:24.572625 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:24.573490 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:24.573599 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:24.577751 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:24.577763 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:24.577770 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:24.577817 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:24.583488 (+ 5671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:24.584504 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:24.584512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:24.584517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:24.584690 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:24.585553 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:24.585557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:24.585559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:24.585888 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:24.586181 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:24.586711 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 11:52:24.586896 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:52:24.589752 22307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:52:24.589866 22307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:52:24.589999 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:24.591795 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:24.600618 22308 catalog_manager.cc:1347] Generated new cluster ID: 79c4fcf5a42247bfa08617a406b3e1ed
I20260924 11:52:24.600752 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:24.616551 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:24.617910 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:24.626356 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc2901863bb4bcd80d5df3e2554d3f9: Generated new TSK 0
I20260924 11:52:24.627135 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:24.636015 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22216
I20260924 11:52:24.637388 22278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:24.729523 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22216
W20260924 11:52:24.746814 22201 connection.cc:535] client connection to 127.0.0.1:45781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45781): Connection reset by peer (error 104)
2026-09-24T11:52:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:24 dist-test-slave-dd32 krb5kdc[22315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:24 dist-test-slave-dd32 krb5kdc[22315](info): set up 2 sockets
Sep 24 11:52:24 dist-test-slave-dd32 krb5kdc[22315](info): commencing operation
krb5kdc: starting...
W20260924 11:52:25.853601 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:26 dist-test-slave-dd32 krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:26.094177 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:26.418350 22328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:26.418722 22328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:26.418900 22328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:26.426136 22328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:26.426306 22328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:26.426419 22328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:26.426528 22328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:26.426618 22328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:26.436434 22328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:26.437768 22328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:26.439075 22328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:26.450763 22336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:26.450829 22334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:26.450809 22333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:26.451390 22328 server_base.cc:1034] running on GCE node
I20260924 11:52:26.452114 22328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:26.453539 22328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:26.454743 22328 hybrid_clock.cc:648] HybridClock initialized: now 1790250746454713 us; error 30 us; skew 500 ppm
I20260924 11:52:26.455085 22328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:26 dist-test-slave-dd32 krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:26.462101 22328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:26.463613 22328 webserver.cc:460] Webserver started at http://127.0.0.1:42327/ using document root <none> and password file <none>
I20260924 11:52:26.464334 22328 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:26.464483 22328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:26.464783 22328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:26.467765 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "1913b0a0e5f0461e85da90e447983a61"
format_stamp: "Formatted at 2026-09-24 11:52:26 on dist-test-slave-dd32"
server_key: "7b64b088b76c162332b60aac5e8c1861"
server_key_iv: "cbece014c37ea03733f0f0883be886a9"
server_key_version: "encryptionkey@0"
I20260924 11:52:26.468505 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "1913b0a0e5f0461e85da90e447983a61"
format_stamp: "Formatted at 2026-09-24 11:52:26 on dist-test-slave-dd32"
server_key: "7b64b088b76c162332b60aac5e8c1861"
server_key_iv: "cbece014c37ea03733f0f0883be886a9"
server_key_version: "encryptionkey@0"
I20260924 11:52:26.473668 22328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 11:52:26.477001 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:26.478248 22328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:26.478435 22328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "1913b0a0e5f0461e85da90e447983a61"
format_stamp: "Formatted at 2026-09-24 11:52:26 on dist-test-slave-dd32"
server_key: "7b64b088b76c162332b60aac5e8c1861"
server_key_iv: "cbece014c37ea03733f0f0883be886a9"
server_key_version: "encryptionkey@0"
I20260924 11:52:26.478626 22328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:26.520807 22328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:26.524932 22328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:26.525285 22328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:26.544251 22328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39889
I20260924 11:52:26.544297 22394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39889 every 8 connection(s)
I20260924 11:52:26.545702 22328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:26.547075 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22328
I20260924 11:52:26.547281 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:26.547799 1864 external_mini_cluster.cc:1428] Setting key 514e9aa29d463c09189c208674a6324b
I20260924 11:52:26.549975 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:26.561358 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:26.549856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43325 (local address 127.0.0.1:39889)
0924 11:52:26.550972 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:26.550984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:26.551024 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:52:26.552574 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:26.552594 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:26.552979 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:26.553360 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:26.553878 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:26.555140 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:26.557027 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:26.557709 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:26.557841 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:26.559120 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:26.559128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:26.559144 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:26.559666 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:26.559740 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:26.559776 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:26.559892 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:26.560188 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:52:26.560381 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":642,"threads_started":1}
I20260924 11:52:26.562781 22395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61: Bootstrap starting.
I20260924 11:52:26.565099 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:26.566053 22395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:26.567018 22395 log.cc:824] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:26.569892 22395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61: No bootstrap required, opened a new log
I20260924 11:52:26.576381 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:26.569310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37697 (local address 127.0.0.1:39889)
0924 11:52:26.569513 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:26.569519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:26.570090 (+ 571us) server_negotiation.cc:407] Connection header received
0924 11:52:26.570301 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:26.570305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:26.570364 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:26.570441 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:26.570995 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:26.571778 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:26.573841 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:26.574291 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:26.574404 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:26.575581 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:26.575585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:26.575589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:26.575604 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:26.575644 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:26.575658 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:26.575747 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:26.575945 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 11:52:26.576016 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:52:26.579440 22395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1913b0a0e5f0461e85da90e447983a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260924 11:52:26.579790 22395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:26.579902 22395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1913b0a0e5f0461e85da90e447983a61, State: Initialized, Role: FOLLOWER
I20260924 11:52:26.580438 22395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1913b0a0e5f0461e85da90e447983a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260924 11:52:26.580659 22395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:26.580803 22395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:26.580961 22395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:26.583192 22395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1913b0a0e5f0461e85da90e447983a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260924 11:52:26.583760 22395 leader_election.cc:302] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1913b0a0e5f0461e85da90e447983a61; no voters:
I20260924 11:52:26.584385 22395 leader_election.cc:288] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:26.584527 22405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:26.585258 22405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [term 1 LEADER]: Becoming Leader. State: Replica: 1913b0a0e5f0461e85da90e447983a61, State: Running, Role: LEADER
I20260924 11:52:26.585806 22405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1913b0a0e5f0461e85da90e447983a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260924 11:52:26.586520 22395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:26.588418 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1913b0a0e5f0461e85da90e447983a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1913b0a0e5f0461e85da90e447983a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } } }
I20260924 11:52:26.588793 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:26.591228 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1913b0a0e5f0461e85da90e447983a61. Latest consensus state: current_term: 1 leader_uuid: "1913b0a0e5f0461e85da90e447983a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1913b0a0e5f0461e85da90e447983a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } } }
I20260924 11:52:26.591614 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:26.592322 22411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:26.599066 22411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:26.608008 22411 catalog_manager.cc:1347] Generated new cluster ID: 4709e0f946fd4433a22a6e42e572abb4
I20260924 11:52:26.608129 22411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:26.618464 22411 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:26.620005 22411 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:26.633867 22411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1913b0a0e5f0461e85da90e447983a61: Generated new TSK 0
I20260924 11:52:26.635018 22411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:26.651314 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22328
W20260924 11:52:26.652890 22390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43325
W20260924 11:52:26.653693 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:26.653889 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22328
W20260924 11:52:26.672847 22313 connection.cc:535] client connection to 127.0.0.1:39889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39889): Connection reset by peer (error 104)
2026-09-24T11:52:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:26 dist-test-slave-dd32 krb5kdc[22426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:26 dist-test-slave-dd32 krb5kdc[22426](info): set up 2 sockets
Sep 24 11:52:26 dist-test-slave-dd32 krb5kdc[22426](info): commencing operation
krb5kdc: starting...
W20260924 11:52:27.769435 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:27 dist-test-slave-dd32 krb5kdc[22426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:28.022507 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:28.349218 22439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:28.349601 22439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:28.349772 22439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:28.357267 22439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:28.357414 22439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:28.357517 22439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:28.357645 22439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:28.357728 22439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:28.367733 22439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:28.369089 22439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:28.370414 22439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:28.382288 22447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:28.382356 22444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:28.382331 22445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:28.383667 22439 server_base.cc:1034] running on GCE node
I20260924 11:52:28.384449 22439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:28.385843 22439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:28.387072 22439 hybrid_clock.cc:648] HybridClock initialized: now 1790250748387035 us; error 24 us; skew 500 ppm
I20260924 11:52:28.387408 22439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:28 dist-test-slave-dd32 krb5kdc[22426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:28.390479 22439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:28.391925 22439 webserver.cc:460] Webserver started at http://127.0.0.1:37763/ using document root <none> and password file <none>
I20260924 11:52:28.392583 22439 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:28.392732 22439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:28.393028 22439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:28.395885 22439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "0ba9635da34345da861edc2e44bcd794"
format_stamp: "Formatted at 2026-09-24 11:52:28 on dist-test-slave-dd32"
server_key: "405dee2e2853f0d4a839c72b32f61f50"
server_key_iv: "a0e49867767d2c48b6a31f5b42141c65"
server_key_version: "encryptionkey@0"
I20260924 11:52:28.396625 22439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "0ba9635da34345da861edc2e44bcd794"
format_stamp: "Formatted at 2026-09-24 11:52:28 on dist-test-slave-dd32"
server_key: "405dee2e2853f0d4a839c72b32f61f50"
server_key_iv: "a0e49867767d2c48b6a31f5b42141c65"
server_key_version: "encryptionkey@0"
I20260924 11:52:28.401641 22439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 11:52:28.404862 22454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:28.406117 22439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:28.406306 22439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "0ba9635da34345da861edc2e44bcd794"
format_stamp: "Formatted at 2026-09-24 11:52:28 on dist-test-slave-dd32"
server_key: "405dee2e2853f0d4a839c72b32f61f50"
server_key_iv: "a0e49867767d2c48b6a31f5b42141c65"
server_key_version: "encryptionkey@0"
I20260924 11:52:28.406502 22439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:28.412062 22439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:28.415779 22439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:28.416059 22439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:28.433900 22439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34723
I20260924 11:52:28.433979 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34723 every 8 connection(s)
I20260924 11:52:28.435241 22439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:28.436332 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22439
I20260924 11:52:28.436511 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:28.437031 1864 external_mini_cluster.cc:1428] Setting key 6a77c4040279dafe8213ed0118dc357a
I20260924 11:52:28.439911 22506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:28.445647 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:28.438935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55461 (local address 127.0.0.1:34723)
0924 11:52:28.439719 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:28.439731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:28.439772 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:52:28.441875 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:28.441897 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:28.442208 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:28.442743 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:28.443915 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:28.443924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:28.443939 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:28.443953 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:28.444022 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:28.444041 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:28.444206 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:28.444413 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 11:52:28.444664 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":352,"threads_started":1}
I20260924 11:52:28.449589 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:28.453929 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794: Bootstrap starting.
I20260924 11:52:28.458225 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:28.455711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38211 (local address 127.0.0.1:34723)
0924 11:52:28.455916 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:28.455922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:28.456220 (+ 298us) server_negotiation.cc:407] Connection header received
0924 11:52:28.456382 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:28.456385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:28.456437 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:28.456526 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:28.457557 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:28.457562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:28.457565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:28.457579 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:28.457620 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:28.457635 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:28.457726 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:28.457864 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 11:52:28.457913 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:52:28.458483 22506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:28.459748 22506 log.cc:824] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:28.462427 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794: No bootstrap required, opened a new log
I20260924 11:52:28.470801 22506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba9635da34345da861edc2e44bcd794" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260924 11:52:28.471309 22506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:28.471441 22506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba9635da34345da861edc2e44bcd794, State: Initialized, Role: FOLLOWER
I20260924 11:52:28.472276 22506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba9635da34345da861edc2e44bcd794" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260924 11:52:28.472550 22506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:28.472712 22506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:28.472895 22506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:28.475632 22506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba9635da34345da861edc2e44bcd794" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260924 11:52:28.476186 22506 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba9635da34345da861edc2e44bcd794; no voters:
I20260924 11:52:28.476766 22506 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:28.476923 22516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:28.478235 22516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba9635da34345da861edc2e44bcd794, State: Running, Role: LEADER
I20260924 11:52:28.478837 22516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba9635da34345da861edc2e44bcd794" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260924 11:52:28.479012 22506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:28.482105 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba9635da34345da861edc2e44bcd794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba9635da34345da861edc2e44bcd794" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } } }
I20260924 11:52:28.482076 22518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba9635da34345da861edc2e44bcd794. Latest consensus state: current_term: 1 leader_uuid: "0ba9635da34345da861edc2e44bcd794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba9635da34345da861edc2e44bcd794" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } } }
I20260924 11:52:28.482540 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:28.482584 22518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:28.483781 22522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:28.490350 22522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:28.499624 22522 catalog_manager.cc:1347] Generated new cluster ID: eab20df08bd54203bccaa6854c8a603c
I20260924 11:52:28.499826 22522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:28.533197 22522 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:28.533679 22522 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:28.545804 22522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba9635da34345da861edc2e44bcd794: Generated new TSK 0
I20260924 11:52:28.546607 22522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:28.557971 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22439
W20260924 11:52:28.559152 22501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55461
W20260924 11:52:28.559806 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:28.559931 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22439
2026-09-24T11:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:28 dist-test-slave-dd32 krb5kdc[22537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:28 dist-test-slave-dd32 krb5kdc[22537](info): set up 2 sockets
Sep 24 11:52:28 dist-test-slave-dd32 krb5kdc[22537](info): commencing operation
krb5kdc: starting...
W20260924 11:52:29.661451 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:29 dist-test-slave-dd32 krb5kdc[22537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:29.920897 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:30.244659 22550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:30.244946 22550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:30.245160 22550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:30.252368 22550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:30.252475 22550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:30.252521 22550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:30.252626 22550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:30.252696 22550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:30.262600 22550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:30.264000 22550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:30.265379 22550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:30.277359 22558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:30.277441 22556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:30.277398 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:30.278414 22550 server_base.cc:1034] running on GCE node
I20260924 11:52:30.279162 22550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:30.280490 22550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:30.281677 22550 hybrid_clock.cc:648] HybridClock initialized: now 1790250750281624 us; error 51 us; skew 500 ppm
I20260924 11:52:30.282015 22550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:30 dist-test-slave-dd32 krb5kdc[22537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:30.285251 22550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:30.286615 22550 webserver.cc:460] Webserver started at http://127.0.0.1:33471/ using document root <none> and password file <none>
I20260924 11:52:30.287300 22550 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:30.287434 22550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:30.287739 22550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:30.290666 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "30084fc3d70644fda0a5f0d59ba8afec"
format_stamp: "Formatted at 2026-09-24 11:52:30 on dist-test-slave-dd32"
server_key: "3f024f80b9b7e224572bb81c939fd55e"
server_key_iv: "6a7e8950001d54a36b98e1180c0d0f15"
server_key_version: "encryptionkey@0"
I20260924 11:52:30.291414 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "30084fc3d70644fda0a5f0d59ba8afec"
format_stamp: "Formatted at 2026-09-24 11:52:30 on dist-test-slave-dd32"
server_key: "3f024f80b9b7e224572bb81c939fd55e"
server_key_iv: "6a7e8950001d54a36b98e1180c0d0f15"
server_key_version: "encryptionkey@0"
I20260924 11:52:30.296593 22550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:52:30.299978 22565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:30.301309 22550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:30.301483 22550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "30084fc3d70644fda0a5f0d59ba8afec"
format_stamp: "Formatted at 2026-09-24 11:52:30 on dist-test-slave-dd32"
server_key: "3f024f80b9b7e224572bb81c939fd55e"
server_key_iv: "6a7e8950001d54a36b98e1180c0d0f15"
server_key_version: "encryptionkey@0"
I20260924 11:52:30.301671 22550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:30.321172 22550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:30.325596 22550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:30.325905 22550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:30.344146 22550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43123
I20260924 11:52:30.344177 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43123 every 8 connection(s)
I20260924 11:52:30.345610 22550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:30.349450 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:30.354943 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22550
I20260924 11:52:30.355145 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:30.355549 1864 external_mini_cluster.cc:1428] Setting key 152865aa939dc80e7d019236b9b5ff74
I20260924 11:52:30.360877 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec: Bootstrap starting.
I20260924 11:52:30.364423 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:30.365542 22617 log.cc:824] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:30.368371 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec: No bootstrap required, opened a new log
Sep 24 11:52:30 dist-test-slave-dd32 krb5kdc[22537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:52:30.378110 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30084fc3d70644fda0a5f0d59ba8afec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43123 } }
I20260924 11:52:30.378537 22617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:30.378643 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30084fc3d70644fda0a5f0d59ba8afec, State: Initialized, Role: FOLLOWER
I20260924 11:52:30.379309 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30084fc3d70644fda0a5f0d59ba8afec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43123 } }
I20260924 11:52:30.379563 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:30.357355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54175 (local address 127.0.0.1:43123)
0924 11:52:30.358231 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:30.358253 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:30.358306 (+ 53us) server_negotiation.cc:407] Connection header received
0924 11:52:30.360638 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:30.360664 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:30.361175 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:30.361565 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:30.362878 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:30.364914 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:30.366423 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:30.367300 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:30.367476 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:30.371214 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:30.371223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:30.371241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:30.371280 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:30.375268 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:30.375888 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:30.375897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:30.375905 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:30.376116 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:30.376470 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:30.376475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:30.376479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:30.377191 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:30.377464 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:30.377853 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 11:52:30.378262 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":380,"threads_started":1}
I20260924 11:52:30.379599 22617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:30.379722 22617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:30.379966 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:30.382993 22617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30084fc3d70644fda0a5f0d59ba8afec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43123 } }
I20260924 11:52:30.383351 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:30.383558 22617 leader_election.cc:302] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30084fc3d70644fda0a5f0d59ba8afec; no voters:
I20260924 11:52:30.384140 22617 leader_election.cc:288] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:30.384274 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:30.384878 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [term 1 LEADER]: Becoming Leader. State: Replica: 30084fc3d70644fda0a5f0d59ba8afec, State: Running, Role: LEADER
I20260924 11:52:30.385506 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30084fc3d70644fda0a5f0d59ba8afec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43123 } }
I20260924 11:52:30.387530 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30084fc3d70644fda0a5f0d59ba8afec. Latest consensus state: current_term: 1 leader_uuid: "30084fc3d70644fda0a5f0d59ba8afec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30084fc3d70644fda0a5f0d59ba8afec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43123 } } }
I20260924 11:52:30.387924 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:30.388432 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30084fc3d70644fda0a5f0d59ba8afec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30084fc3d70644fda0a5f0d59ba8afec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43123 } } }
I20260924 11:52:30.388851 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:30.396898 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:30.397150 22628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:30.403116 22628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:30.418565 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:30.397121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48055 (local address 127.0.0.1:43123)
0924 11:52:30.397310 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:30.397319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:30.397825 (+ 506us) server_negotiation.cc:407] Connection header received
0924 11:52:30.398048 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:30.398053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:30.398129 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:30.398240 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:30.400259 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:30.401495 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:30.403919 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:30.404941 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:30.405094 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:30.409223 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:30.409233 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:30.409240 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:30.409302 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:30.414146 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:30.415305 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:30.415314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:30.415320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:30.415505 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:30.416095 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:30.416104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:30.416109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:30.416643 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:30.416912 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:30.417696 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 11:52:30.417992 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":60}
I20260924 11:52:30.425678 22628 catalog_manager.cc:1347] Generated new cluster ID: 1ee7434795144838a87ba5d157c67724
I20260924 11:52:30.425819 22628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:30.449194 22628 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:30.451068 22628 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:30.462667 22628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30084fc3d70644fda0a5f0d59ba8afec: Generated new TSK 0
I20260924 11:52:30.463794 22628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:30.474721 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22550
I20260924 11:52:30.475991 22612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:30.567554 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22550
2026-09-24T11:52:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:30 dist-test-slave-dd32 krb5kdc[22649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:30 dist-test-slave-dd32 krb5kdc[22649](info): set up 2 sockets
Sep 24 11:52:30 dist-test-slave-dd32 krb5kdc[22649](info): commencing operation
krb5kdc: starting...
W20260924 11:52:31.669431 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:31 dist-test-slave-dd32 krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:31.920508 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:32.245501 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:32.245852 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:32.246021 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:32.253453 22662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:32.253597 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:32.253706 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:32.253814 22662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:32.253888 22662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:32.263548 22662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:32.264848 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:32.266170 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:32.277933 22670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:32.277997 22667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:32.277977 22668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:32.279085 22662 server_base.cc:1034] running on GCE node
I20260924 11:52:32.279860 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:32.281278 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:32.282500 22662 hybrid_clock.cc:648] HybridClock initialized: now 1790250752282466 us; error 44 us; skew 500 ppm
I20260924 11:52:32.282907 22662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:32 dist-test-slave-dd32 krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:32.292397 22662 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:32.294026 22662 webserver.cc:460] Webserver started at http://127.0.0.1:41907/ using document root <none> and password file <none>
I20260924 11:52:32.294714 22662 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:32.294879 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:32.295181 22662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:32.298208 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "68f1bd464b244850a0856e349f69c216"
format_stamp: "Formatted at 2026-09-24 11:52:32 on dist-test-slave-dd32"
server_key: "8142a578a82a5a4cfe55aa6dcdf78436"
server_key_iv: "cbe1762b3e29917b5afb5b853b598482"
server_key_version: "encryptionkey@0"
I20260924 11:52:32.298990 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "68f1bd464b244850a0856e349f69c216"
format_stamp: "Formatted at 2026-09-24 11:52:32 on dist-test-slave-dd32"
server_key: "8142a578a82a5a4cfe55aa6dcdf78436"
server_key_iv: "cbe1762b3e29917b5afb5b853b598482"
server_key_version: "encryptionkey@0"
I20260924 11:52:32.304056 22662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:52:32.307271 22677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:32.308451 22662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 11:52:32.308627 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "68f1bd464b244850a0856e349f69c216"
format_stamp: "Formatted at 2026-09-24 11:52:32 on dist-test-slave-dd32"
server_key: "8142a578a82a5a4cfe55aa6dcdf78436"
server_key_iv: "cbe1762b3e29917b5afb5b853b598482"
server_key_version: "encryptionkey@0"
I20260924 11:52:32.308817 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:32.334458 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:32.338867 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:32.339140 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:32.357497 22662 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35093
I20260924 11:52:32.357533 22728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35093 every 8 connection(s)
I20260924 11:52:32.358898 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:32.362660 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:32.365054 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22662
I20260924 11:52:32.365247 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:32.365753 1864 external_mini_cluster.cc:1428] Setting key ab688f5282007066d47f8047e7ddae1c
I20260924 11:52:32.374818 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216: Bootstrap starting.
I20260924 11:52:32.378307 22729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:32.378899 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:32.367525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53581 (local address 127.0.0.1:35093)
0924 11:52:32.368376 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:32.368390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:32.368435 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:52:32.370159 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:32.370182 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:32.370566 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:32.370878 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:32.371430 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:32.372758 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:32.374580 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:32.375288 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:32.375428 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:32.376598 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:32.376605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:32.376617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:32.377196 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:32.377274 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:32.377298 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:32.377414 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:32.377640 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 11:52:32.377900 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":414,"threads_started":1}
I20260924 11:52:32.379496 22729 log.cc:824] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:32.382509 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216: No bootstrap required, opened a new log
I20260924 11:52:32.382709 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:32.392951 22729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f1bd464b244850a0856e349f69c216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260924 11:52:32.393483 22729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:32.393612 22729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f1bd464b244850a0856e349f69c216, State: Initialized, Role: FOLLOWER
I20260924 11:52:32.394291 22729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f1bd464b244850a0856e349f69c216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260924 11:52:32.394419 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:32.387849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60597 (local address 127.0.0.1:35093)
0924 11:52:32.388007 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:32.388014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:32.388350 (+ 336us) server_negotiation.cc:407] Connection header received
0924 11:52:32.388592 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:32.388598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:32.388686 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:32.388800 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:32.389457 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:32.390135 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:32.391951 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:32.392389 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:32.392479 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:32.393656 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:32.393661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:32.393665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:32.393681 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:32.393739 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:32.393760 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:32.393840 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:32.393995 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 11:52:32.394092 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 11:52:32.394558 22729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:32.394708 22729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:32.394901 22729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:32.397122 22729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f1bd464b244850a0856e349f69c216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260924 11:52:32.397595 22729 leader_election.cc:302] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f1bd464b244850a0856e349f69c216; no voters:
I20260924 11:52:32.398132 22729 leader_election.cc:288] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:32.398243 22739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:32.399005 22739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [term 1 LEADER]: Becoming Leader. State: Replica: 68f1bd464b244850a0856e349f69c216, State: Running, Role: LEADER
I20260924 11:52:32.399487 22739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f1bd464b244850a0856e349f69c216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260924 11:52:32.400035 22729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:32.402164 22741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f1bd464b244850a0856e349f69c216. Latest consensus state: current_term: 1 leader_uuid: "68f1bd464b244850a0856e349f69c216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f1bd464b244850a0856e349f69c216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } } }
I20260924 11:52:32.402523 22741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:32.403299 22740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f1bd464b244850a0856e349f69c216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f1bd464b244850a0856e349f69c216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } } }
I20260924 11:52:32.403548 22740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:32.404122 22746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:32.409556 22746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:32.419102 22746 catalog_manager.cc:1347] Generated new cluster ID: 3eed309d2851499b85dc8b7cd3a8ce00
I20260924 11:52:32.419222 22746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:32.436744 22746 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:32.438139 22746 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:32.450538 22746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f1bd464b244850a0856e349f69c216: Generated new TSK 0
I20260924 11:52:32.451416 22746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:32.463649 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22662
W20260924 11:52:32.464766 22724 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53581
W20260924 11:52:32.465507 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:32.465672 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22662
2026-09-24T11:52:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:32 dist-test-slave-dd32 krb5kdc[22760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:32 dist-test-slave-dd32 krb5kdc[22760](info): set up 2 sockets
Sep 24 11:52:32 dist-test-slave-dd32 krb5kdc[22760](info): commencing operation
krb5kdc: starting...
W20260924 11:52:33.557606 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:33 dist-test-slave-dd32 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:33.809151 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:34.136103 22773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:34.136385 22773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:34.136580 22773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:34.143707 22773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:34.143823 22773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:34.143909 22773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:34.144019 22773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:34.144097 22773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:34.154160 22773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:34.155505 22773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:34.156829 22773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:34.168363 22781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:34.168484 22779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:34.168478 22778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:34.169453 22773 server_base.cc:1034] running on GCE node
I20260924 11:52:34.170166 22773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:34.171422 22773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:34.172639 22773 hybrid_clock.cc:648] HybridClock initialized: now 1790250754172599 us; error 54 us; skew 500 ppm
I20260924 11:52:34.172969 22773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:34 dist-test-slave-dd32 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:34.176033 22773 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:34.177421 22773 webserver.cc:460] Webserver started at http://127.0.0.1:40515/ using document root <none> and password file <none>
I20260924 11:52:34.178108 22773 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:34.178252 22773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:34.178551 22773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:34.181421 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "cb0cbbd9822547e095eabb5c9524ed50"
format_stamp: "Formatted at 2026-09-24 11:52:34 on dist-test-slave-dd32"
server_key: "1e56ebad7941b769f26ab97be3eda1b2"
server_key_iv: "64c72420ff45c789d158ac949e7c4bad"
server_key_version: "encryptionkey@0"
I20260924 11:52:34.182169 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "cb0cbbd9822547e095eabb5c9524ed50"
format_stamp: "Formatted at 2026-09-24 11:52:34 on dist-test-slave-dd32"
server_key: "1e56ebad7941b769f26ab97be3eda1b2"
server_key_iv: "64c72420ff45c789d158ac949e7c4bad"
server_key_version: "encryptionkey@0"
I20260924 11:52:34.186990 22773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:52:34.190284 22788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:34.191473 22773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:34.191648 22773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "cb0cbbd9822547e095eabb5c9524ed50"
format_stamp: "Formatted at 2026-09-24 11:52:34 on dist-test-slave-dd32"
server_key: "1e56ebad7941b769f26ab97be3eda1b2"
server_key_iv: "64c72420ff45c789d158ac949e7c4bad"
server_key_version: "encryptionkey@0"
I20260924 11:52:34.191835 22773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:34.197132 22773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:34.201354 22773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:34.201651 22773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:34.219830 22773 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42909
I20260924 11:52:34.219877 22839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42909 every 8 connection(s)
I20260924 11:52:34.221306 22773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:34.223995 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22773
I20260924 11:52:34.224175 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:34.224730 1864 external_mini_cluster.cc:1428] Setting key 347cc187536b9d43d8409351c9c78b98
I20260924 11:52:34.225346 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:34.233564 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:34.227318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52477 (local address 127.0.0.1:42909)
0924 11:52:34.228125 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:34.228136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:34.228175 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:52:34.230202 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:34.230223 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:34.230568 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:34.230992 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:34.232008 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:34.232014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:34.232025 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:34.232038 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:34.232087 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:34.232104 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:34.232342 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:34.232488 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 11:52:34.232652 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":346,"threads_started":1}
I20260924 11:52:34.237030 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:34.238489 22840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50: Bootstrap starting.
I20260924 11:52:34.241729 22840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:34.243074 22840 log.cc:824] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:34.244023 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:34.241299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42243 (local address 127.0.0.1:42909)
0924 11:52:34.241491 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:34.241499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:34.242022 (+ 523us) server_negotiation.cc:407] Connection header received
0924 11:52:34.242195 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:34.242199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:34.242244 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:34.242321 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:34.243274 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:34.243278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:34.243282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:34.243299 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:34.243339 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:34.243354 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:34.243438 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:34.243629 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 11:52:34.243694 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:52:34.246117 22840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50: No bootstrap required, opened a new log
I20260924 11:52:34.253295 22840 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb0cbbd9822547e095eabb5c9524ed50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260924 11:52:34.253767 22840 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:34.253887 22840 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0cbbd9822547e095eabb5c9524ed50, State: Initialized, Role: FOLLOWER
I20260924 11:52:34.254428 22840 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb0cbbd9822547e095eabb5c9524ed50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260924 11:52:34.254657 22840 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:34.254801 22840 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:34.254978 22840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:34.257228 22840 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb0cbbd9822547e095eabb5c9524ed50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260924 11:52:34.257761 22840 leader_election.cc:302] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb0cbbd9822547e095eabb5c9524ed50; no voters:
I20260924 11:52:34.258332 22840 leader_election.cc:288] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:34.258473 22850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:34.259202 22850 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [term 1 LEADER]: Becoming Leader. State: Replica: cb0cbbd9822547e095eabb5c9524ed50, State: Running, Role: LEADER
I20260924 11:52:34.260001 22850 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb0cbbd9822547e095eabb5c9524ed50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } }
I20260924 11:52:34.260639 22840 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:34.262879 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb0cbbd9822547e095eabb5c9524ed50. Latest consensus state: current_term: 1 leader_uuid: "cb0cbbd9822547e095eabb5c9524ed50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb0cbbd9822547e095eabb5c9524ed50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } } }
I20260924 11:52:34.263123 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb0cbbd9822547e095eabb5c9524ed50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb0cbbd9822547e095eabb5c9524ed50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42909 } } }
I20260924 11:52:34.263346 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:34.263373 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:34.265090 22858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:34.270898 22858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:34.280040 22858 catalog_manager.cc:1347] Generated new cluster ID: c0ab327231e942f5b8e515fa1f136a9a
I20260924 11:52:34.280181 22858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:34.298945 22858 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:34.299404 22858 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:34.311476 22858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb0cbbd9822547e095eabb5c9524ed50: Generated new TSK 0
I20260924 11:52:34.312588 22858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:34.325070 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22773
W20260924 11:52:34.326421 22835 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52477
W20260924 11:52:34.327178 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22773: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:34.327298 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22773
2026-09-24T11:52:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:34 dist-test-slave-dd32 krb5kdc[22871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:34 dist-test-slave-dd32 krb5kdc[22871](info): set up 2 sockets
Sep 24 11:52:34 dist-test-slave-dd32 krb5kdc[22871](info): commencing operation
krb5kdc: starting...
W20260924 11:52:35.429929 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:35 dist-test-slave-dd32 krb5kdc[22871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:35.680060 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:36.023931 22884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:36.024214 22884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:36.024379 22884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:36.031509 22884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:36.031615 22884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:36.031754 22884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:36.031853 22884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:36.031934 22884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:36.041646 22884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:36.042876 22884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:36.044166 22884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:36.055873 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:36.056041 22892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:36.056376 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:36.056866 22884 server_base.cc:1034] running on GCE node
I20260924 11:52:36.057693 22884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:36.059056 22884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:36.060318 22884 hybrid_clock.cc:648] HybridClock initialized: now 1790250756060237 us; error 61 us; skew 500 ppm
I20260924 11:52:36.060672 22884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:36 dist-test-slave-dd32 krb5kdc[22871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:36.063767 22884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:36.065227 22884 webserver.cc:460] Webserver started at http://127.0.0.1:38939/ using document root <none> and password file <none>
I20260924 11:52:36.065864 22884 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:36.065961 22884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:36.066196 22884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:36.069131 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "ba64b64512f24578a9582f2ceaa84663"
format_stamp: "Formatted at 2026-09-24 11:52:36 on dist-test-slave-dd32"
server_key: "9c6d895c1e9e7a4463daadecb985517d"
server_key_iv: "6a9c1b38539783e181ae137a369a57ca"
server_key_version: "encryptionkey@0"
I20260924 11:52:36.069840 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "ba64b64512f24578a9582f2ceaa84663"
format_stamp: "Formatted at 2026-09-24 11:52:36 on dist-test-slave-dd32"
server_key: "9c6d895c1e9e7a4463daadecb985517d"
server_key_iv: "6a9c1b38539783e181ae137a369a57ca"
server_key_version: "encryptionkey@0"
I20260924 11:52:36.075080 22884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:52:36.078713 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:36.079967 22884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:52:36.080153 22884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "ba64b64512f24578a9582f2ceaa84663"
format_stamp: "Formatted at 2026-09-24 11:52:36 on dist-test-slave-dd32"
server_key: "9c6d895c1e9e7a4463daadecb985517d"
server_key_iv: "6a9c1b38539783e181ae137a369a57ca"
server_key_version: "encryptionkey@0"
I20260924 11:52:36.080358 22884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:36.099000 22884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:36.102891 22884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:36.103168 22884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:36.121552 22884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44147
I20260924 11:52:36.121585 22950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44147 every 8 connection(s)
I20260924 11:52:36.122889 22884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:36.125499 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22884
I20260924 11:52:36.125684 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:36.126192 1864 external_mini_cluster.cc:1428] Setting key b647a37634b4506e49f087c693af7b57
I20260924 11:52:36.127031 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:36.138499 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663: Bootstrap starting.
Sep 24 11:52:36 dist-test-slave-dd32 krb5kdc[22871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:52:36.141700 22951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:36.142729 22951 log.cc:824] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:36.145495 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663: No bootstrap required, opened a new log
I20260924 11:52:36.147259 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:36.127960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:58619 (local address 127.0.0.1:44147)
0924 11:52:36.128680 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:36.128692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:36.128732 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:52:36.130281 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:36.130301 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:36.130694 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:36.131085 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:36.132513 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:36.133888 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:36.135131 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:36.135806 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:36.136022 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:36.140036 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:36.140043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:36.140056 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:36.140094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:36.143212 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:36.144008 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:36.144014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:36.144020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:36.144184 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:36.144678 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:36.144682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:36.144684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:36.145214 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:36.145545 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:36.145997 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 11:52:36.146324 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260924 11:52:36.151306 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:36.154069 22951 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba64b64512f24578a9582f2ceaa84663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260924 11:52:36.154495 22951 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:36.154625 22951 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba64b64512f24578a9582f2ceaa84663, State: Initialized, Role: FOLLOWER
I20260924 11:52:36.155296 22951 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba64b64512f24578a9582f2ceaa84663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260924 11:52:36.155535 22951 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:36.155689 22951 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:36.155900 22951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:36.158492 22951 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba64b64512f24578a9582f2ceaa84663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260924 11:52:36.159121 22951 leader_election.cc:302] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba64b64512f24578a9582f2ceaa84663; no voters:
I20260924 11:52:36.159749 22951 leader_election.cc:288] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:36.159893 22961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:36.161829 22951 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:36.165381 22961 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [term 1 LEADER]: Becoming Leader. State: Replica: ba64b64512f24578a9582f2ceaa84663, State: Running, Role: LEADER
I20260924 11:52:36.166105 22961 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba64b64512f24578a9582f2ceaa84663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260924 11:52:36.169412 22963 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba64b64512f24578a9582f2ceaa84663. Latest consensus state: current_term: 1 leader_uuid: "ba64b64512f24578a9582f2ceaa84663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba64b64512f24578a9582f2ceaa84663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } } }
I20260924 11:52:36.170051 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba64b64512f24578a9582f2ceaa84663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba64b64512f24578a9582f2ceaa84663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } } }
I20260924 11:52:36.169875 22963 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:36.170310 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:36.177297 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:36.155919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:55649 (local address 127.0.0.1:44147)
0924 11:52:36.156112 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:36.156121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:36.156599 (+ 478us) server_negotiation.cc:407] Connection header received
0924 11:52:36.156719 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:36.156724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:36.156813 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:36.156923 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:36.158415 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:36.159569 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:36.161945 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:36.162628 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:36.162766 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:36.166618 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:36.166628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:36.166636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:36.166687 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:36.171855 (+ 5168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:36.173216 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:36.173224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:36.173230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:36.173467 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:36.174214 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:36.174220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:36.174224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:36.174699 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:36.176235 (+ 1536us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:36.176470 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 11:52:36.176705 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":60}
I20260924 11:52:36.183228 22971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:36.193472 22971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:36.204005 22971 catalog_manager.cc:1347] Generated new cluster ID: e7128fbf70724860a8ebdaee00a455ed
I20260924 11:52:36.204120 22971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:36.215070 22971 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:36.216873 22971 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:36.228674 22971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba64b64512f24578a9582f2ceaa84663: Generated new TSK 0
I20260924 11:52:36.229710 22971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:36.243912 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22884
I20260924 11:52:36.245714 22946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:36.336135 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22884
W20260924 11:52:36.352918 22869 connection.cc:535] client connection to 127.0.0.1:44147 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44147): Connection reset by peer (error 104)
2026-09-24T11:52:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:36 dist-test-slave-dd32 krb5kdc[22984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:36 dist-test-slave-dd32 krb5kdc[22984](info): set up 2 sockets
Sep 24 11:52:36 dist-test-slave-dd32 krb5kdc[22984](info): commencing operation
krb5kdc: starting...
W20260924 11:52:37.437623 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:37 dist-test-slave-dd32 krb5kdc[22984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:37.682557 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:38.018775 22997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:38.019090 22997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:38.019289 22997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:38.027624 22997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:38.027793 22997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:38.027909 22997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:38.028018 22997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:38.028112 22997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:38.039338 22997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:38.040748 22997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:38.042157 22997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:38.054430 23005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:38.054522 23003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:38.054749 23002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:38.055534 22997 server_base.cc:1034] running on GCE node
I20260924 11:52:38.056322 22997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:38.057726 22997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:38.058928 22997 hybrid_clock.cc:648] HybridClock initialized: now 1790250758058894 us; error 34 us; skew 500 ppm
I20260924 11:52:38.059278 22997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:38 dist-test-slave-dd32 krb5kdc[22984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:38.067613 22997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:38.069147 22997 webserver.cc:460] Webserver started at http://127.0.0.1:46867/ using document root <none> and password file <none>
I20260924 11:52:38.069796 22997 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:38.069947 22997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:38.070236 22997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:38.073155 22997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "5a8fef8630af46c78ee80e0c173ec149"
format_stamp: "Formatted at 2026-09-24 11:52:38 on dist-test-slave-dd32"
server_key: "e21737bf2673b01ba3fa4adb28c8702b"
server_key_iv: "eade2a6b94c09dde8200583c5e1ee266"
server_key_version: "encryptionkey@0"
I20260924 11:52:38.073925 22997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "5a8fef8630af46c78ee80e0c173ec149"
format_stamp: "Formatted at 2026-09-24 11:52:38 on dist-test-slave-dd32"
server_key: "e21737bf2673b01ba3fa4adb28c8702b"
server_key_iv: "eade2a6b94c09dde8200583c5e1ee266"
server_key_version: "encryptionkey@0"
I20260924 11:52:38.078804 22997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 11:52:38.082156 23012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:38.083402 22997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:52:38.083583 22997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "5a8fef8630af46c78ee80e0c173ec149"
format_stamp: "Formatted at 2026-09-24 11:52:38 on dist-test-slave-dd32"
server_key: "e21737bf2673b01ba3fa4adb28c8702b"
server_key_iv: "eade2a6b94c09dde8200583c5e1ee266"
server_key_version: "encryptionkey@0"
I20260924 11:52:38.083776 22997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:38.098963 22997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:38.102793 22997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:38.103070 22997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:38.121181 22997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35217
I20260924 11:52:38.121214 23063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35217 every 8 connection(s)
I20260924 11:52:38.122597 22997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:38.126209 23064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:38.127658 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 22997
I20260924 11:52:38.127852 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:38.128273 1864 external_mini_cluster.cc:1428] Setting key c83d1d950c599a3189d060f102e25a01
I20260924 11:52:38.139987 23064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149: Bootstrap starting.
I20260924 11:52:38.141510 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:38.130043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:40259 (local address 127.0.0.1:35217)
0924 11:52:38.130836 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:38.130847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:38.130895 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:52:38.132489 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:38.132513 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:38.132915 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:38.133321 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:38.133903 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:38.135141 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:38.137025 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:38.137762 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:38.137897 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:38.139179 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:38.139186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:38.139199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:38.139714 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:38.139785 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:38.139812 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:38.139935 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:38.140311 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 11:52:38.140505 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":360,"threads_started":1}
I20260924 11:52:38.143138 23064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:38.144249 23064 log.cc:824] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:38.145179 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:38.147071 23064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149: No bootstrap required, opened a new log
I20260924 11:52:38.155983 23064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fef8630af46c78ee80e0c173ec149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 11:52:38.156486 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:38.149767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:48611 (local address 127.0.0.1:35217)
0924 11:52:38.149947 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:38.149953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:38.150335 (+ 382us) server_negotiation.cc:407] Connection header received
0924 11:52:38.150499 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:38.150505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:38.150583 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:38.150694 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:38.151143 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:38.152336 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:38.153844 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:38.154553 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:38.154688 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:38.155745 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:38.155752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:38.155760 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:38.155782 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:38.155825 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:38.155844 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:38.155959 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:38.156121 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 11:52:38.156170 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:52:38.156500 23064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:38.156636 23064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8fef8630af46c78ee80e0c173ec149, State: Initialized, Role: FOLLOWER
I20260924 11:52:38.157337 23064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fef8630af46c78ee80e0c173ec149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 11:52:38.157585 23064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:38.157747 23064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:38.157946 23064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:38.160068 23064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fef8630af46c78ee80e0c173ec149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 11:52:38.160542 23064 leader_election.cc:302] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a8fef8630af46c78ee80e0c173ec149; no voters:
I20260924 11:52:38.161160 23064 leader_election.cc:288] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:38.161302 23074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:38.162653 23074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [term 1 LEADER]: Becoming Leader. State: Replica: 5a8fef8630af46c78ee80e0c173ec149, State: Running, Role: LEADER
I20260924 11:52:38.163165 23074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fef8630af46c78ee80e0c173ec149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 11:52:38.163439 23064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:38.166319 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a8fef8630af46c78ee80e0c173ec149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fef8630af46c78ee80e0c173ec149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260924 11:52:38.166266 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a8fef8630af46c78ee80e0c173ec149. Latest consensus state: current_term: 1 leader_uuid: "5a8fef8630af46c78ee80e0c173ec149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fef8630af46c78ee80e0c173ec149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260924 11:52:38.166728 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:38.166762 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:38.167948 23083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:38.173454 23083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:38.181839 23083 catalog_manager.cc:1347] Generated new cluster ID: 2be43b027b90421da9816233ef95b68c
I20260924 11:52:38.181945 23083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:38.228964 23083 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:38.230409 23083 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:38.236876 23083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a8fef8630af46c78ee80e0c173ec149: Generated new TSK 0
I20260924 11:52:38.237890 23083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:38.248083 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22997
W20260924 11:52:38.249221 23059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.190:40259
W20260924 11:52:38.249993 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 22997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:38.250137 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 22997
W20260924 11:52:38.263844 22982 connection.cc:535] client connection to 127.0.0.1:35217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35217): Connection reset by peer (error 104)
2026-09-24T11:52:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:38 dist-test-slave-dd32 krb5kdc[23095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:38 dist-test-slave-dd32 krb5kdc[23095](info): set up 2 sockets
Sep 24 11:52:38 dist-test-slave-dd32 krb5kdc[23095](info): commencing operation
krb5kdc: starting...
W20260924 11:52:39.345516 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:39 dist-test-slave-dd32 krb5kdc[23095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:39.590521 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:39.933283 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:39.933553 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:39.933738 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:39.940793 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:39.940900 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:39.941018 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:39.941118 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:39.941200 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:39.950766 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:39.951998 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:39.953336 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:39.964915 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:39.964952 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:39.965107 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:39.965907 23108 server_base.cc:1034] running on GCE node
I20260924 11:52:39.966635 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:39.967922 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:39.969163 23108 hybrid_clock.cc:648] HybridClock initialized: now 1790250759969129 us; error 46 us; skew 500 ppm
I20260924 11:52:39.969504 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:39 dist-test-slave-dd32 krb5kdc[23095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:39.972432 23108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:39.974035 23108 webserver.cc:460] Webserver started at http://127.0.0.1:41607/ using document root <none> and password file <none>
I20260924 11:52:39.974701 23108 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:39.974848 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:39.975138 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:39.978037 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "44668621030c4e9e9feaad11db156725"
format_stamp: "Formatted at 2026-09-24 11:52:39 on dist-test-slave-dd32"
server_key: "22e76b66b820b1684190409d3efa2057"
server_key_iv: "8a5aa293a7cfc4a52e0cd439d8b764ca"
server_key_version: "encryptionkey@0"
I20260924 11:52:39.978749 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "44668621030c4e9e9feaad11db156725"
format_stamp: "Formatted at 2026-09-24 11:52:39 on dist-test-slave-dd32"
server_key: "22e76b66b820b1684190409d3efa2057"
server_key_iv: "8a5aa293a7cfc4a52e0cd439d8b764ca"
server_key_version: "encryptionkey@0"
I20260924 11:52:39.983708 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:52:39.986984 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:39.988271 23108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:52:39.988476 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "44668621030c4e9e9feaad11db156725"
format_stamp: "Formatted at 2026-09-24 11:52:39 on dist-test-slave-dd32"
server_key: "22e76b66b820b1684190409d3efa2057"
server_key_iv: "8a5aa293a7cfc4a52e0cd439d8b764ca"
server_key_version: "encryptionkey@0"
I20260924 11:52:39.988672 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:39.994329 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:39.998055 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:39.998340 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:40.016644 23108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38779
I20260924 11:52:40.016678 23174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38779 every 8 connection(s)
I20260924 11:52:40.018107 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:40.021854 23175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:40.024822 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23108
I20260924 11:52:40.025007 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:40.025549 1864 external_mini_cluster.cc:1428] Setting key 08cd414c920a9b426bba6ab714d00a7d
I20260924 11:52:40.033320 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:40.027380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:54225 (local address 127.0.0.1:38779)
0924 11:52:40.028137 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:40.028148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:40.028187 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:52:40.029903 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:40.029925 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:40.030261 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:40.030658 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:40.031714 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:40.031720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:40.031750 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:40.031764 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:40.031827 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:40.031845 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:40.032098 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:40.032228 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 11:52:40.032402 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":324,"threads_started":1}
I20260924 11:52:40.034180 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725: Bootstrap starting.
I20260924 11:52:40.036895 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:40.037617 23175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:40.038599 23175 log.cc:824] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:40.041510 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725: No bootstrap required, opened a new log
I20260924 11:52:40.043810 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:40.041369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:56141 (local address 127.0.0.1:38779)
0924 11:52:40.041559 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:40.041564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:40.041894 (+ 330us) server_negotiation.cc:407] Connection header received
0924 11:52:40.042073 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:40.042077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:40.042125 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:40.042201 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:40.043165 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:40.043169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:40.043172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:40.043185 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:40.043224 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:40.043241 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:40.043329 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:40.043461 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 11:52:40.043509 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:52:40.049858 23175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44668621030c4e9e9feaad11db156725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38779 } }
I20260924 11:52:40.050222 23175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:40.050335 23175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44668621030c4e9e9feaad11db156725, State: Initialized, Role: FOLLOWER
I20260924 11:52:40.050962 23175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44668621030c4e9e9feaad11db156725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38779 } }
I20260924 11:52:40.051226 23175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:40.051370 23175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:40.051507 23175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:40.053638 23175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44668621030c4e9e9feaad11db156725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38779 } }
I20260924 11:52:40.054050 23175 leader_election.cc:302] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44668621030c4e9e9feaad11db156725; no voters:
I20260924 11:52:40.054528 23175 leader_election.cc:288] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:40.054670 23185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:40.055299 23185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [term 1 LEADER]: Becoming Leader. State: Replica: 44668621030c4e9e9feaad11db156725, State: Running, Role: LEADER
I20260924 11:52:40.055819 23185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44668621030c4e9e9feaad11db156725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38779 } }
I20260924 11:52:40.056488 23175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:40.058851 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44668621030c4e9e9feaad11db156725. Latest consensus state: current_term: 1 leader_uuid: "44668621030c4e9e9feaad11db156725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44668621030c4e9e9feaad11db156725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38779 } } }
I20260924 11:52:40.059223 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:40.058431 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44668621030c4e9e9feaad11db156725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44668621030c4e9e9feaad11db156725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38779 } } }
I20260924 11:52:40.059538 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:40.060940 23193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:40.066080 23193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:40.076442 23193 catalog_manager.cc:1347] Generated new cluster ID: 18f13044c6484e8daa339c3e405b099f
I20260924 11:52:40.076555 23193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:40.086063 23193 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:40.086539 23193 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:40.104176 23193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44668621030c4e9e9feaad11db156725: Generated new TSK 0
I20260924 11:52:40.105255 23193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:40.123210 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23108
W20260924 11:52:40.124230 23170 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.190:54225
W20260924 11:52:40.124933 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23108: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:40.125105 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23108
2026-09-24T11:52:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:40 dist-test-slave-dd32 krb5kdc[23206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:40 dist-test-slave-dd32 krb5kdc[23206](info): set up 2 sockets
Sep 24 11:52:40 dist-test-slave-dd32 krb5kdc[23206](info): commencing operation
krb5kdc: starting...
W20260924 11:52:41.225417 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:41 dist-test-slave-dd32 krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:41.475471 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:41.800897 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:41.801251 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:41.801429 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:41.808706 23219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:41.808859 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:41.808966 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:41.809094 23219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:41.809172 23219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:41.818925 23219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44541
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:41.820222 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:41.821575 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:41.834692 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:41.834718 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:41.835912 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:41.836015 23219 server_base.cc:1034] running on GCE node
I20260924 11:52:41.836922 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:41.838322 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:41.839552 23219 hybrid_clock.cc:648] HybridClock initialized: now 1790250761839514 us; error 54 us; skew 500 ppm
I20260924 11:52:41.839895 23219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:41 dist-test-slave-dd32 krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:41.842953 23219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:41.844440 23219 webserver.cc:460] Webserver started at http://127.0.0.1:39447/ using document root <none> and password file <none>
I20260924 11:52:41.845221 23219 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:41.845369 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:41.845655 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:41.848973 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "8170eb54b0aa49c3b8080eb2e7b688a6"
format_stamp: "Formatted at 2026-09-24 11:52:41 on dist-test-slave-dd32"
server_key: "0e6d6c1427adfa6fffe3dca295a5fd09"
server_key_iv: "31ec5ecb4dc84f43b938039b8c58ccb1"
server_key_version: "encryptionkey@0"
I20260924 11:52:41.849771 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "8170eb54b0aa49c3b8080eb2e7b688a6"
format_stamp: "Formatted at 2026-09-24 11:52:41 on dist-test-slave-dd32"
server_key: "0e6d6c1427adfa6fffe3dca295a5fd09"
server_key_iv: "31ec5ecb4dc84f43b938039b8c58ccb1"
server_key_version: "encryptionkey@0"
I20260924 11:52:41.854653 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:52:41.857941 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:41.859155 23219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:41.859339 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "8170eb54b0aa49c3b8080eb2e7b688a6"
format_stamp: "Formatted at 2026-09-24 11:52:41 on dist-test-slave-dd32"
server_key: "0e6d6c1427adfa6fffe3dca295a5fd09"
server_key_iv: "31ec5ecb4dc84f43b938039b8c58ccb1"
server_key_version: "encryptionkey@0"
I20260924 11:52:41.859532 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:41.895076 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:41.898984 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:41.899287 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:41.918195 23219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44541
I20260924 11:52:41.918215 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44541 every 8 connection(s)
I20260924 11:52:41.919636 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:41.920003 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23219
I20260924 11:52:41.920166 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:41.920719 1864 external_mini_cluster.cc:1428] Setting key 2447463e0d87d045d5c9f688bf8fd723
I20260924 11:52:41.924084 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:52:41 dist-test-slave-dd32 krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:52:41.934803 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6: Bootstrap starting.
I20260924 11:52:41.938191 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:41.939225 23286 log.cc:824] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:41.942076 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6: No bootstrap required, opened a new log
I20260924 11:52:41.942204 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:41.922608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:50175 (local address 127.0.0.1:44541)
0924 11:52:41.923344 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:41.923356 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:41.923398 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:52:41.925165 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:41.925186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:41.925556 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:41.925940 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:41.927384 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:41.928669 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:41.929936 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:41.930644 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:41.930833 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:41.935154 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:41.935162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:41.935175 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:41.935215 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:41.938187 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:41.938968 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:41.938975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:41.938980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:41.939140 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:41.939640 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:41.939645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:41.939647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:41.940147 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:41.940505 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:41.940955 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 11:52:41.941278 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":317,"threads_started":1}
I20260924 11:52:41.946048 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:41.950534 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44541 } }
I20260924 11:52:41.951009 23286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:41.951133 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8170eb54b0aa49c3b8080eb2e7b688a6, State: Initialized, Role: FOLLOWER
I20260924 11:52:41.951711 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44541 } }
I20260924 11:52:41.951912 23286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:41.952081 23286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:41.952250 23286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:41.954964 23286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44541 } }
I20260924 11:52:41.955528 23286 leader_election.cc:302] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8170eb54b0aa49c3b8080eb2e7b688a6; no voters:
I20260924 11:52:41.956176 23286 leader_election.cc:288] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:41.956293 23296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:41.961277 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:41.961556 23296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8170eb54b0aa49c3b8080eb2e7b688a6, State: Running, Role: LEADER
I20260924 11:52:41.962153 23296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44541 } }
I20260924 11:52:41.965025 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44541 } } }
I20260924 11:52:41.965413 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:41.965902 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8170eb54b0aa49c3b8080eb2e7b688a6. Latest consensus state: current_term: 1 leader_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8170eb54b0aa49c3b8080eb2e7b688a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44541 } } }
I20260924 11:52:41.966223 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:41.968689 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:41.950178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:51497 (local address 127.0.0.1:44541)
0924 11:52:41.950316 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:41.950320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:41.950775 (+ 455us) server_negotiation.cc:407] Connection header received
0924 11:52:41.950898 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:41.950903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:41.950987 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:41.951099 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:41.952653 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:41.953897 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:41.955116 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:41.955767 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:41.955911 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:41.959457 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:41.959464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:41.959468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:41.959503 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:41.963506 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:41.964754 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:41.964762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:41.964769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:41.965027 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:41.965762 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:41.965770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:41.965776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:41.966272 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:41.967750 (+ 1478us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:41.967964 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 11:52:41.968185 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 11:52:41.974076 23304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:41.979447 23304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:41.987689 23304 catalog_manager.cc:1347] Generated new cluster ID: b7a940e5cca648aab06690720ffb9e6d
I20260924 11:52:41.987818 23304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:42.002178 23304 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:42.003644 23304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:42.011024 23304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8170eb54b0aa49c3b8080eb2e7b688a6: Generated new TSK 0
I20260924 11:52:42.011905 23304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:42.020123 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23219
I20260924 11:52:42.021184 23281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:42.109967 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23219
W20260924 11:52:42.126700 23204 connection.cc:535] client connection to 127.0.0.1:44541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44541): Connection reset by peer (error 104)
2026-09-24T11:52:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:42 dist-test-slave-dd32 krb5kdc[23319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:42 dist-test-slave-dd32 krb5kdc[23319](info): set up 2 sockets
Sep 24 11:52:42 dist-test-slave-dd32 krb5kdc[23319](info): commencing operation
krb5kdc: starting...
W20260924 11:52:43.209491 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:43 dist-test-slave-dd32 krb5kdc[23319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:43.456409 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:43.779332 23332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:43.779610 23332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:43.779807 23332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:43.787083 23332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:43.787217 23332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:43.787305 23332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:43.787410 23332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:43.787492 23332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:43.797245 23332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:43.798550 23332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:43.799860 23332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:43.811206 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:43.811149 23338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:43.811120 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:43.812165 23332 server_base.cc:1034] running on GCE node
I20260924 11:52:43.812883 23332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:43.814288 23332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:43.815514 23332 hybrid_clock.cc:648] HybridClock initialized: now 1790250763815475 us; error 52 us; skew 500 ppm
I20260924 11:52:43.815850 23332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:43 dist-test-slave-dd32 krb5kdc[23319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:43.823366 23332 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:43.824869 23332 webserver.cc:460] Webserver started at http://127.0.0.1:36077/ using document root <none> and password file <none>
I20260924 11:52:43.825598 23332 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:43.825742 23332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:43.826048 23332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:43.828953 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "030868aedbb34974bb76788eab4244aa"
format_stamp: "Formatted at 2026-09-24 11:52:43 on dist-test-slave-dd32"
server_key: "264cde045c01c5e3527c9559a79aca03"
server_key_iv: "0b41f433e6c76ad044b971ab946d1d66"
server_key_version: "encryptionkey@0"
I20260924 11:52:43.829804 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "030868aedbb34974bb76788eab4244aa"
format_stamp: "Formatted at 2026-09-24 11:52:43 on dist-test-slave-dd32"
server_key: "264cde045c01c5e3527c9559a79aca03"
server_key_iv: "0b41f433e6c76ad044b971ab946d1d66"
server_key_version: "encryptionkey@0"
I20260924 11:52:43.834726 23332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:52:43.838021 23347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:43.839262 23332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:52:43.839452 23332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "030868aedbb34974bb76788eab4244aa"
format_stamp: "Formatted at 2026-09-24 11:52:43 on dist-test-slave-dd32"
server_key: "264cde045c01c5e3527c9559a79aca03"
server_key_iv: "0b41f433e6c76ad044b971ab946d1d66"
server_key_version: "encryptionkey@0"
I20260924 11:52:43.839668 23332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:43.861428 23332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:43.865772 23332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:43.866071 23332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:43.884184 23332 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43181
I20260924 11:52:43.884218 23398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43181 every 8 connection(s)
I20260924 11:52:43.885636 23332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:43.889263 23399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:43.891044 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23332
I20260924 11:52:43.891253 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:43.891703 1864 external_mini_cluster.cc:1428] Setting key 0c66f42e762befc97856bf738db0e029
I20260924 11:52:43.903518 23399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa: Bootstrap starting.
I20260924 11:52:43.906124 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:43.893608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:36423 (local address 127.0.0.1:43181)
0924 11:52:43.894412 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:43.894428 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:43.894487 (+ 59us) server_negotiation.cc:407] Connection header received
0924 11:52:43.896204 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:43.896233 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:43.896757 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:43.897194 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:43.897670 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:43.899584 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:43.900930 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:43.901956 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:43.902138 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:43.903294 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:43.903305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:43.903323 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:43.904053 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:43.904148 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:43.904178 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:43.904332 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:43.904585 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 11:52:43.904752 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":341,"threads_started":1}
I20260924 11:52:43.907325 23399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:43.908399 23399 log.cc:824] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:43.909970 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:43.911500 23399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa: No bootstrap required, opened a new log
I20260924 11:52:43.920959 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:43.914522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:48693 (local address 127.0.0.1:43181)
0924 11:52:43.914713 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:43.914718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:43.915041 (+ 323us) server_negotiation.cc:407] Connection header received
0924 11:52:43.915176 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:43.915181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:43.915253 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:43.915351 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:43.915805 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:43.916976 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:43.918312 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:43.919024 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:43.919156 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:43.920193 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:43.920201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:43.920207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:43.920229 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:43.920288 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:43.920307 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:43.920438 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:43.920590 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 11:52:43.920648 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:52:43.920647 23399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030868aedbb34974bb76788eab4244aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43181 } }
I20260924 11:52:43.921135 23399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:43.921279 23399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030868aedbb34974bb76788eab4244aa, State: Initialized, Role: FOLLOWER
I20260924 11:52:43.921897 23399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030868aedbb34974bb76788eab4244aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43181 } }
I20260924 11:52:43.922155 23399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:43.922307 23399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:43.922485 23399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:43.924583 23399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030868aedbb34974bb76788eab4244aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43181 } }
I20260924 11:52:43.925114 23399 leader_election.cc:302] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 030868aedbb34974bb76788eab4244aa; no voters:
I20260924 11:52:43.925673 23399 leader_election.cc:288] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:43.925872 23409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:43.926570 23409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [term 1 LEADER]: Becoming Leader. State: Replica: 030868aedbb34974bb76788eab4244aa, State: Running, Role: LEADER
I20260924 11:52:43.927093 23409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030868aedbb34974bb76788eab4244aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43181 } }
I20260924 11:52:43.927754 23399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:43.930042 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 030868aedbb34974bb76788eab4244aa. Latest consensus state: current_term: 1 leader_uuid: "030868aedbb34974bb76788eab4244aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030868aedbb34974bb76788eab4244aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43181 } } }
I20260924 11:52:43.930408 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:43.931281 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "030868aedbb34974bb76788eab4244aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030868aedbb34974bb76788eab4244aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43181 } } }
I20260924 11:52:43.932278 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:43.934500 23416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:43.939791 23416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:43.948714 23416 catalog_manager.cc:1347] Generated new cluster ID: 9cc865fde8f2446f85d70ce284ec5bd4
I20260924 11:52:43.948825 23416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:43.963040 23416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:43.964596 23416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:43.970942 23416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 030868aedbb34974bb76788eab4244aa: Generated new TSK 0
I20260924 11:52:43.971762 23416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:43.977137 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23332
W20260924 11:52:43.978190 23394 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.190:36423
W20260924 11:52:43.978936 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23332: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:43.979090 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23332
W20260924 11:52:43.995322 23317 connection.cc:535] client connection to 127.0.0.1:43181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43181): Connection reset by peer (error 104)
2026-09-24T11:52:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:44 dist-test-slave-dd32 krb5kdc[23430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:44 dist-test-slave-dd32 krb5kdc[23430](info): set up 2 sockets
Sep 24 11:52:44 dist-test-slave-dd32 krb5kdc[23430](info): commencing operation
krb5kdc: starting...
W20260924 11:52:45.077590 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:45 dist-test-slave-dd32 krb5kdc[23430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:52:45.327450 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:45.651846 23443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:45.652181 23443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:45.652347 23443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:45.659479 23443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:45.659631 23443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:45.659732 23443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:45.659845 23443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:45.659924 23443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:45.669425 23443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:45.670728 23443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:45.672017 23443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:45.684049 23451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:45.684154 23448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:45.684150 23449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:45.684985 23443 server_base.cc:1034] running on GCE node
I20260924 11:52:45.685729 23443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:45.687057 23443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:45.688259 23443 hybrid_clock.cc:648] HybridClock initialized: now 1790250765688213 us; error 33 us; skew 500 ppm
I20260924 11:52:45.688614 23443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:45 dist-test-slave-dd32 krb5kdc[23430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:45.691704 23443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:52:45.693184 23443 webserver.cc:460] Webserver started at http://127.0.0.1:45159/ using document root <none> and password file <none>
I20260924 11:52:45.693830 23443 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:45.693959 23443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:45.694258 23443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:45.697217 23443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "d40a732785d7446ba5cc8ff071daca39"
format_stamp: "Formatted at 2026-09-24 11:52:45 on dist-test-slave-dd32"
server_key: "146b5787f3dd5862e3bb56afb0d09037"
server_key_iv: "df57209f096ebcf69dc6e9726b5575b5"
server_key_version: "encryptionkey@0"
I20260924 11:52:45.697952 23443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "d40a732785d7446ba5cc8ff071daca39"
format_stamp: "Formatted at 2026-09-24 11:52:45 on dist-test-slave-dd32"
server_key: "146b5787f3dd5862e3bb56afb0d09037"
server_key_iv: "df57209f096ebcf69dc6e9726b5575b5"
server_key_version: "encryptionkey@0"
I20260924 11:52:45.703039 23443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 11:52:45.706517 23458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:45.707772 23443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:52:45.708132 23443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "d40a732785d7446ba5cc8ff071daca39"
format_stamp: "Formatted at 2026-09-24 11:52:45 on dist-test-slave-dd32"
server_key: "146b5787f3dd5862e3bb56afb0d09037"
server_key_iv: "df57209f096ebcf69dc6e9726b5575b5"
server_key_version: "encryptionkey@0"
I20260924 11:52:45.708377 23443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:45.714229 23443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:45.718537 23443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:45.718833 23443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:45.737203 23443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33775
I20260924 11:52:45.737239 23509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33775 every 8 connection(s)
I20260924 11:52:45.738656 23443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:45.741055 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23443
I20260924 11:52:45.741284 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:45.741714 1864 external_mini_cluster.cc:1428] Setting key 3e417dadd9f77248c9917c859afaba1d
I20260924 11:52:45.742635 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:45.750205 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:45.743589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:51761 (local address 127.0.0.1:33775)
0924 11:52:45.744371 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:45.744387 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:45.744436 (+ 49us) server_negotiation.cc:407] Connection header received
0924 11:52:45.746226 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:45.746253 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:45.746710 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:45.747172 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:45.748164 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:45.748172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:45.748190 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:45.748214 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:45.748301 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:45.748327 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:45.748544 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:45.748681 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 11:52:45.748902 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
I20260924 11:52:45.753834 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:45.757759 23510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39: Bootstrap starting.
I20260924 11:52:45.761693 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:45.759162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:44575 (local address 127.0.0.1:33775)
0924 11:52:45.759345 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:45.759351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:45.759715 (+ 364us) server_negotiation.cc:407] Connection header received
0924 11:52:45.759848 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:45.759854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:45.759931 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:45.760056 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:45.760991 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:45.760996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:45.761000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:45.761016 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:45.761056 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:45.761095 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:45.761201 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:45.761296 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 11:52:45.761370 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:52:45.761693 23510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:45.762905 23510 log.cc:824] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:45.765582 23510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39: No bootstrap required, opened a new log
I20260924 11:52:45.772532 23510 raft_consensus.cc:348] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40a732785d7446ba5cc8ff071daca39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260924 11:52:45.772899 23510 raft_consensus.cc:374] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:45.773002 23510 raft_consensus.cc:729] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40a732785d7446ba5cc8ff071daca39, State: Initialized, Role: FOLLOWER
I20260924 11:52:45.773559 23510 consensus_queue.cc:260] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40a732785d7446ba5cc8ff071daca39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260924 11:52:45.773788 23510 raft_consensus.cc:388] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:45.773942 23510 raft_consensus.cc:482] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:45.774127 23510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:45.776840 23510 raft_consensus.cc:504] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40a732785d7446ba5cc8ff071daca39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260924 11:52:45.777427 23510 leader_election.cc:302] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d40a732785d7446ba5cc8ff071daca39; no voters:
I20260924 11:52:45.778165 23510 leader_election.cc:288] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:45.778350 23520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:45.779067 23520 raft_consensus.cc:686] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [term 1 LEADER]: Becoming Leader. State: Replica: d40a732785d7446ba5cc8ff071daca39, State: Running, Role: LEADER
I20260924 11:52:45.779605 23520 consensus_queue.cc:237] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40a732785d7446ba5cc8ff071daca39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260924 11:52:45.780438 23510 sys_catalog.cc:564] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:45.782982 23521 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d40a732785d7446ba5cc8ff071daca39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40a732785d7446ba5cc8ff071daca39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } } }
I20260924 11:52:45.783370 23521 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:45.783906 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d40a732785d7446ba5cc8ff071daca39. Latest consensus state: current_term: 1 leader_uuid: "d40a732785d7446ba5cc8ff071daca39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40a732785d7446ba5cc8ff071daca39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } } }
I20260924 11:52:45.784222 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:45.785106 23528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:45.790086 23528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:45.799229 23528 catalog_manager.cc:1347] Generated new cluster ID: 5e1914d8fd3f4849bc55771bb38e1fcd
I20260924 11:52:45.799360 23528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:45.819087 23528 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:45.819562 23528 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:45.824525 23528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d40a732785d7446ba5cc8ff071daca39: Generated new TSK 0
I20260924 11:52:45.825392 23528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:45.835638 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23443
W20260924 11:52:45.836858 23505 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.190:51761
W20260924 11:52:45.837736 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23443: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:45.837880 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23443
2026-09-24T11:52:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:45 dist-test-slave-dd32 krb5kdc[23541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:45 dist-test-slave-dd32 krb5kdc[23541](info): set up 2 sockets
Sep 24 11:52:45 dist-test-slave-dd32 krb5kdc[23541](info): commencing operation
krb5kdc: starting...
W20260924 11:52:46.949393 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:47 dist-test-slave-dd32 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:47.195977 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41119
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:41119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:47.520895 23554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:47.521246 23554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:47.521415 23554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:47.528545 23554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:47.528690 23554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:47.528797 23554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:47.528906 23554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:47.528993 23554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:47.538621 23554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:41119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:47.540259 23554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:47.541707 23554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:47.553556 23559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:47.553615 23560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:47.553639 23562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:47.554263 23554 server_base.cc:1034] running on GCE node
I20260924 11:52:47.554965 23554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:47.556245 23554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:47.557471 23554 hybrid_clock.cc:648] HybridClock initialized: now 1790250767557433 us; error 47 us; skew 500 ppm
I20260924 11:52:47.557801 23554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:47 dist-test-slave-dd32 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250767, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:47.560707 23554 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:47.562088 23554 webserver.cc:460] Webserver started at http://127.1.210.62:44023/ using document root <none> and password file <none>
I20260924 11:52:47.562738 23554 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:47.562881 23554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:47.563176 23554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:47.566336 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "c77d9937ae7e4192a6f28bc74da95bf6"
format_stamp: "Formatted at 2026-09-24 11:52:47 on dist-test-slave-dd32"
server_key: "664da8b8b2539811ec326603ceebda6c"
server_key_iv: "d8e6e1db248613200763f17297c57fbe"
server_key_version: "encryptionkey@0"
I20260924 11:52:47.567171 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "c77d9937ae7e4192a6f28bc74da95bf6"
format_stamp: "Formatted at 2026-09-24 11:52:47 on dist-test-slave-dd32"
server_key: "664da8b8b2539811ec326603ceebda6c"
server_key_iv: "d8e6e1db248613200763f17297c57fbe"
server_key_version: "encryptionkey@0"
I20260924 11:52:47.572093 23554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:52:47.575346 23569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:47.576603 23554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:52:47.576803 23554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "c77d9937ae7e4192a6f28bc74da95bf6"
format_stamp: "Formatted at 2026-09-24 11:52:47 on dist-test-slave-dd32"
server_key: "664da8b8b2539811ec326603ceebda6c"
server_key_iv: "d8e6e1db248613200763f17297c57fbe"
server_key_version: "encryptionkey@0"
I20260924 11:52:47.576992 23554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:47.595743 23554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:47.599701 23554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:47.600020 23554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:47.618662 23554 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:41119
I20260924 11:52:47.618693 23620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:41119 every 8 connection(s)
I20260924 11:52:47.620033 23554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:47.623615 23621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:47.627873 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23554
I20260924 11:52:47.628077 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:47.628499 1864 external_mini_cluster.cc:1428] Setting key 4c6782929879b23bc6184c29e4c1f046
I20260924 11:52:47.634394 23621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6: Bootstrap starting.
I20260924 11:52:47.637866 23621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:47.638984 23621 log.cc:824] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:47.642122 23621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6: No bootstrap required, opened a new log
Sep 24 11:52:47 dist-test-slave-dd32 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:47.650205 23621 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41119 } }
I20260924 11:52:47.650743 23621 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:47.650872 23621 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77d9937ae7e4192a6f28bc74da95bf6, State: Initialized, Role: FOLLOWER
I20260924 11:52:47.651629 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:47.630242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47877 (local address 127.1.210.62:41119)
0924 11:52:47.631018 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:47.631032 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:47.631071 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:52:47.632766 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:47.632788 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:47.633213 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:47.633518 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:47.635275 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:47.636569 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:47.638406 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:47.639025 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:47.639199 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:47.643932 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:47.643941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:47.643954 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:47.643996 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:47.647208 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:47.648009 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:47.648016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:47.648021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:47.648186 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:47.648685 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:47.648689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:47.648691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:47.649293 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:47.649663 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:47.650117 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 11:52:47.650456 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":358,"threads_started":1}
I20260924 11:52:47.651548 23621 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41119 } }
I20260924 11:52:47.651809 23621 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:47.651991 23621 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:47.652174 23621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:47.654888 23621 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41119 } }
I20260924 11:52:47.655390 23621 leader_election.cc:302] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77d9937ae7e4192a6f28bc74da95bf6; no voters:
I20260924 11:52:47.655684 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:47.656071 23621 leader_election.cc:288] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:47.656196 23626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:47.658089 23621 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:47.661604 23626 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [term 1 LEADER]: Becoming Leader. State: Replica: c77d9937ae7e4192a6f28bc74da95bf6, State: Running, Role: LEADER
I20260924 11:52:47.662374 23626 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41119 } }
I20260924 11:52:47.665746 23627 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41119 } } }
I20260924 11:52:47.666208 23627 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:47.666793 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77d9937ae7e4192a6f28bc74da95bf6. Latest consensus state: current_term: 1 leader_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d9937ae7e4192a6f28bc74da95bf6" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41119 } } }
I20260924 11:52:47.667101 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:47.667104 23636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:47.677028 23636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:47.692441 23636 catalog_manager.cc:1347] Generated new cluster ID: 0b0cfe73f9fa4d7aa7b064ac19bad6f9
I20260924 11:52:47.692644 23636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:47.699352 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:47.674769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55271 (local address 127.1.210.62:41119)
0924 11:52:47.674937 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:47.674943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:47.681567 (+ 6624us) server_negotiation.cc:407] Connection header received
0924 11:52:47.681832 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:47.681838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:47.681918 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:47.682000 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:47.683976 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:47.685101 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:47.687299 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:47.688217 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:47.688336 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:47.692037 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:47.692045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:47.692050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:47.692093 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:47.695942 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:47.696879 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:47.696885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:47.696888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:47.697051 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:47.697603 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:47.697608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:47.697612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:47.697989 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:47.698225 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:47.698833 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 11:52:47.699026 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:52:47.718955 23636 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:47.720863 23636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:47.737743 23636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77d9937ae7e4192a6f28bc74da95bf6: Generated new TSK 0
I20260924 11:52:47.738812 23636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:47.746603 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23554
I20260924 11:52:47.748750 23616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:47.839591 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23554
W20260924 11:52:47.856626 23539 connection.cc:535] client connection to 127.1.210.62:41119 recv error: Network error: failed to read from TLS socket (remote: 127.1.210.62:41119): Connection reset by peer (error 104)
2026-09-24T11:52:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:47 dist-test-slave-dd32 krb5kdc[23654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:47 dist-test-slave-dd32 krb5kdc[23654](info): set up 2 sockets
Sep 24 11:52:47 dist-test-slave-dd32 krb5kdc[23654](info): commencing operation
krb5kdc: starting...
W20260924 11:52:48.942130 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:49 dist-test-slave-dd32 krb5kdc[23654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:49.202669 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43349
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:49.533661 23667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:49.533975 23667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:49.534183 23667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:49.541389 23667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:49.541527 23667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:49.541615 23667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:49.541720 23667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:49.541781 23667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:49.551628 23667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:49.553081 23667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:49.554411 23667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:49.566785 23673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:49.566815 23672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:49.567085 23675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:49.567732 23667 server_base.cc:1034] running on GCE node
I20260924 11:52:49.568590 23667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:49.570043 23667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:49.571270 23667 hybrid_clock.cc:648] HybridClock initialized: now 1790250769571216 us; error 42 us; skew 500 ppm
I20260924 11:52:49.571615 23667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:49 dist-test-slave-dd32 krb5kdc[23654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250769, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:49.578485 23667 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:49.580022 23667 webserver.cc:460] Webserver started at http://127.1.210.62:43633/ using document root <none> and password file <none>
I20260924 11:52:49.580726 23667 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:49.580878 23667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:49.581245 23667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:49.584270 23667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "850f67e747d24bb68d34ad0e4ae70821"
format_stamp: "Formatted at 2026-09-24 11:52:49 on dist-test-slave-dd32"
server_key: "37d2eff06a6238ffcb6544b1239a02d6"
server_key_iv: "b09f22c1e41f688fd5975b150acbe65e"
server_key_version: "encryptionkey@0"
I20260924 11:52:49.585049 23667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "850f67e747d24bb68d34ad0e4ae70821"
format_stamp: "Formatted at 2026-09-24 11:52:49 on dist-test-slave-dd32"
server_key: "37d2eff06a6238ffcb6544b1239a02d6"
server_key_iv: "b09f22c1e41f688fd5975b150acbe65e"
server_key_version: "encryptionkey@0"
I20260924 11:52:49.590207 23667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:52:49.593616 23682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:49.594868 23667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:52:49.595062 23667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "850f67e747d24bb68d34ad0e4ae70821"
format_stamp: "Formatted at 2026-09-24 11:52:49 on dist-test-slave-dd32"
server_key: "37d2eff06a6238ffcb6544b1239a02d6"
server_key_iv: "b09f22c1e41f688fd5975b150acbe65e"
server_key_version: "encryptionkey@0"
I20260924 11:52:49.595257 23667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:49.617048 23667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:49.621006 23667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:49.621309 23667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:49.639658 23667 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43349
I20260924 11:52:49.639689 23733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43349 every 8 connection(s)
I20260924 11:52:49.640962 23667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:49.644552 23734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:49.647405 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23667
I20260924 11:52:49.647591 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:49.648119 1864 external_mini_cluster.cc:1428] Setting key 1df8c5da404812d5e14f6e9b09b028fc
I20260924 11:52:49.656392 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821: Bootstrap starting.
I20260924 11:52:49.660035 23734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:49.661237 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:49.649918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58275 (local address 127.1.210.62:43349)
0924 11:52:49.650715 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:49.650726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:49.650766 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:52:49.652390 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:49.652411 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:49.652800 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:49.653190 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:49.653722 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:49.655031 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:49.656832 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:49.657561 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:49.657683 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:49.658966 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:49.658972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:49.658985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:49.659485 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:49.659542 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:49.659564 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:49.659676 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:49.659964 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 11:52:49.660225 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260924 11:52:49.661217 23734 log.cc:824] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:49.664199 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821: No bootstrap required, opened a new log
I20260924 11:52:49.665036 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:49.672521 23734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850f67e747d24bb68d34ad0e4ae70821" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43349 } }
I20260924 11:52:49.672878 23734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:49.673007 23734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850f67e747d24bb68d34ad0e4ae70821, State: Initialized, Role: FOLLOWER
I20260924 11:52:49.673606 23734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850f67e747d24bb68d34ad0e4ae70821" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43349 } }
I20260924 11:52:49.673872 23734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:49.674057 23734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:49.674283 23734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:49.676025 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:49.669257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54463 (local address 127.1.210.62:43349)
0924 11:52:49.669446 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:49.669452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:49.669862 (+ 410us) server_negotiation.cc:407] Connection header received
0924 11:52:49.669994 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:49.669999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:49.670084 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:49.670188 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:49.670659 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:49.671831 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:49.673213 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:49.673746 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:49.673893 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:49.675239 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:49.675244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:49.675256 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:49.675275 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:49.675334 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:49.675353 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:49.675489 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:49.675683 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 11:52:49.675745 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:52:49.677357 23734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850f67e747d24bb68d34ad0e4ae70821" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43349 } }
I20260924 11:52:49.677943 23734 leader_election.cc:302] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 850f67e747d24bb68d34ad0e4ae70821; no voters:
I20260924 11:52:49.678439 23734 leader_election.cc:288] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:49.678596 23744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:49.679272 23744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [term 1 LEADER]: Becoming Leader. State: Replica: 850f67e747d24bb68d34ad0e4ae70821, State: Running, Role: LEADER
I20260924 11:52:49.679814 23744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850f67e747d24bb68d34ad0e4ae70821" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43349 } }
I20260924 11:52:49.680817 23734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:49.682961 23745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "850f67e747d24bb68d34ad0e4ae70821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850f67e747d24bb68d34ad0e4ae70821" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43349 } } }
I20260924 11:52:49.682933 23746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 850f67e747d24bb68d34ad0e4ae70821. Latest consensus state: current_term: 1 leader_uuid: "850f67e747d24bb68d34ad0e4ae70821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850f67e747d24bb68d34ad0e4ae70821" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43349 } } }
I20260924 11:52:49.683383 23745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:49.683420 23746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:49.685485 23751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:49.690896 23751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:49.700030 23751 catalog_manager.cc:1347] Generated new cluster ID: 7faae30848e3435285583f2ec39f4355
I20260924 11:52:49.700153 23751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:49.717423 23751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:49.718962 23751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:49.725901 23751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 850f67e747d24bb68d34ad0e4ae70821: Generated new TSK 0
I20260924 11:52:49.726761 23751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:49.741895 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23667
W20260924 11:52:49.743031 23729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58275
W20260924 11:52:49.743809 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:49.743937 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23667
W20260924 11:52:49.760844 23652 connection.cc:535] client connection to 127.1.210.62:43349 recv error: Network error: failed to read from TLS socket (remote: 127.1.210.62:43349): Connection reset by peer (error 104)
2026-09-24T11:52:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:49 dist-test-slave-dd32 krb5kdc[23765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:49 dist-test-slave-dd32 krb5kdc[23765](info): set up 2 sockets
Sep 24 11:52:49 dist-test-slave-dd32 krb5kdc[23765](info): commencing operation
krb5kdc: starting...
W20260924 11:52:50.862032 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:51 dist-test-slave-dd32 krb5kdc[23765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:51.115973 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43725
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:51.438519 23778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:51.438884 23778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:51.439064 23778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:51.446272 23778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:51.446424 23778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:51.446530 23778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:51.446636 23778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:51.446712 23778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:51.456434 23778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:43725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:51.457767 23778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:51.459158 23778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:51.471069 23786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:51.471031 23783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:51.471104 23784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:51.472285 23778 server_base.cc:1034] running on GCE node
I20260924 11:52:51.473030 23778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:51.474407 23778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:51.475643 23778 hybrid_clock.cc:648] HybridClock initialized: now 1790250771475604 us; error 35 us; skew 500 ppm
I20260924 11:52:51.475987 23778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:51 dist-test-slave-dd32 krb5kdc[23765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250771, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:51.479059 23778 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:51.480377 23778 webserver.cc:460] Webserver started at http://127.1.210.62:39723/ using document root <none> and password file <none>
I20260924 11:52:51.480981 23778 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:51.481133 23778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:51.481391 23778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:51.484165 23778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "f7ae931eed9f46d8892c0604de974a32"
format_stamp: "Formatted at 2026-09-24 11:52:51 on dist-test-slave-dd32"
server_key: "dbc1d12e284db215e55cf0473c068396"
server_key_iv: "44625af4beb97bf26c6d980e36ce78d4"
server_key_version: "encryptionkey@0"
I20260924 11:52:51.484875 23778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "f7ae931eed9f46d8892c0604de974a32"
format_stamp: "Formatted at 2026-09-24 11:52:51 on dist-test-slave-dd32"
server_key: "dbc1d12e284db215e55cf0473c068396"
server_key_iv: "44625af4beb97bf26c6d980e36ce78d4"
server_key_version: "encryptionkey@0"
I20260924 11:52:51.489809 23778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 11:52:51.492962 23793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:51.494249 23778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:52:51.494385 23778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "f7ae931eed9f46d8892c0604de974a32"
format_stamp: "Formatted at 2026-09-24 11:52:51 on dist-test-slave-dd32"
server_key: "dbc1d12e284db215e55cf0473c068396"
server_key_iv: "44625af4beb97bf26c6d980e36ce78d4"
server_key_version: "encryptionkey@0"
I20260924 11:52:51.494551 23778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:51.500118 23778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:51.503835 23778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:51.504065 23778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:51.521962 23778 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:43725
I20260924 11:52:51.522008 23844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:43725 every 8 connection(s)
I20260924 11:52:51.523346 23778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:51.526968 23845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:51.529867 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23778
I20260924 11:52:51.530064 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:51.530496 1864 external_mini_cluster.cc:1428] Setting key f1ebfb040267983fcf76da6d162ca9bc
I20260924 11:52:51.538504 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:51.532313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60091 (local address 127.1.210.62:43725)
0924 11:52:51.533379 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:51.533391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:51.533458 (+ 67us) server_negotiation.cc:407] Connection header received
0924 11:52:51.535127 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:51.535148 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:51.535527 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:51.535849 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:51.536956 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:51.536962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:51.536974 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:51.536990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:51.537050 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:51.537113 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:51.537287 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:51.537421 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 11:52:51.537636 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":597,"threads_started":1}
I20260924 11:52:51.539177 23845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32: Bootstrap starting.
I20260924 11:52:51.542477 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:51.542989 23845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:51.544008 23845 log.cc:824] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:51.547143 23845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32: No bootstrap required, opened a new log
I20260924 11:52:51.549829 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:51.547070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33865 (local address 127.1.210.62:43725)
0924 11:52:51.547290 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:51.547295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:51.547778 (+ 483us) server_negotiation.cc:407] Connection header received
0924 11:52:51.547975 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:51.547980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:51.548038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:51.548139 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:51.549134 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:51.549138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:51.549141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:51.549158 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:51.549193 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:51.549207 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:51.549339 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:51.549507 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 11:52:51.549575 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 11:52:51.555365 23845 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ae931eed9f46d8892c0604de974a32" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43725 } }
I20260924 11:52:51.555780 23845 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:51.555925 23845 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ae931eed9f46d8892c0604de974a32, State: Initialized, Role: FOLLOWER
I20260924 11:52:51.556563 23845 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ae931eed9f46d8892c0604de974a32" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43725 } }
I20260924 11:52:51.556807 23845 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:51.556957 23845 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:51.557169 23845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:51.559232 23845 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ae931eed9f46d8892c0604de974a32" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43725 } }
I20260924 11:52:51.559721 23845 leader_election.cc:302] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ae931eed9f46d8892c0604de974a32; no voters:
I20260924 11:52:51.560405 23845 leader_election.cc:288] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:51.560546 23855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:51.561204 23855 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [term 1 LEADER]: Becoming Leader. State: Replica: f7ae931eed9f46d8892c0604de974a32, State: Running, Role: LEADER
I20260924 11:52:51.561789 23855 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ae931eed9f46d8892c0604de974a32" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43725 } }
I20260924 11:52:51.562681 23845 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:51.564714 23857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ae931eed9f46d8892c0604de974a32. Latest consensus state: current_term: 1 leader_uuid: "f7ae931eed9f46d8892c0604de974a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ae931eed9f46d8892c0604de974a32" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43725 } } }
I20260924 11:52:51.564723 23856 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ae931eed9f46d8892c0604de974a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ae931eed9f46d8892c0604de974a32" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 43725 } } }
I20260924 11:52:51.565205 23857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:51.565222 23856 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:51.566943 23860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:51.571904 23860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:51.580279 23860 catalog_manager.cc:1347] Generated new cluster ID: 2180bea589bf44b0ac9a2616caf37f14
I20260924 11:52:51.580384 23860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:51.621007 23860 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:51.621533 23860 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:51.641054 23860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ae931eed9f46d8892c0604de974a32: Generated new TSK 0
I20260924 11:52:51.641897 23860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:51.651150 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23778
W20260924 11:52:51.652186 23840 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60091
W20260924 11:52:51.652877 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23778: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:51.653031 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23778
2026-09-24T11:52:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:51 dist-test-slave-dd32 krb5kdc[23876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:51 dist-test-slave-dd32 krb5kdc[23876](info): set up 2 sockets
Sep 24 11:52:51 dist-test-slave-dd32 krb5kdc[23876](info): commencing operation
krb5kdc: starting...
W20260924 11:52:52.757495 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:52 dist-test-slave-dd32 krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:53.014516 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42089
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:53.336717 23889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:53.337088 23889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:53.337260 23889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:53.344507 23889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:53.344687 23889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:53.344785 23889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:53.344877 23889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:53.344944 23889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:53.354676 23889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:42089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:53.356022 23889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:53.357429 23889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:53.368989 23895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:53.369230 23897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:53.369525 23894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:53.369855 23889 server_base.cc:1034] running on GCE node
I20260924 11:52:53.370639 23889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:53.371959 23889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:53.373155 23889 hybrid_clock.cc:648] HybridClock initialized: now 1790250773373095 us; error 48 us; skew 500 ppm
I20260924 11:52:53.373484 23889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:53 dist-test-slave-dd32 krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:53.376578 23889 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:53.377978 23889 webserver.cc:460] Webserver started at http://127.1.210.62:37757/ using document root <none> and password file <none>
I20260924 11:52:53.378657 23889 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:53.378788 23889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:53.379092 23889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:53.382054 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "e15ff2cdedb44c2ea59a403a70ca9718"
format_stamp: "Formatted at 2026-09-24 11:52:53 on dist-test-slave-dd32"
server_key: "38f7e26b8d7c6b7c8bb795b758f316e3"
server_key_iv: "d2ee35a20910933576d85dddfdadd262"
server_key_version: "encryptionkey@0"
I20260924 11:52:53.382788 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "e15ff2cdedb44c2ea59a403a70ca9718"
format_stamp: "Formatted at 2026-09-24 11:52:53 on dist-test-slave-dd32"
server_key: "38f7e26b8d7c6b7c8bb795b758f316e3"
server_key_iv: "d2ee35a20910933576d85dddfdadd262"
server_key_version: "encryptionkey@0"
I20260924 11:52:53.387769 23889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:52:53.391016 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:53.392271 23889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:52:53.392443 23889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "e15ff2cdedb44c2ea59a403a70ca9718"
format_stamp: "Formatted at 2026-09-24 11:52:53 on dist-test-slave-dd32"
server_key: "38f7e26b8d7c6b7c8bb795b758f316e3"
server_key_iv: "d2ee35a20910933576d85dddfdadd262"
server_key_version: "encryptionkey@0"
I20260924 11:52:53.392633 23889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:53.406510 23889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:53.410143 23889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:53.410434 23889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:53.428349 23889 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:42089
I20260924 11:52:53.428385 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:42089 every 8 connection(s)
I20260924 11:52:53.429730 23889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:53.433337 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:53.436898 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 23889
I20260924 11:52:53.437112 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:53.437628 1864 external_mini_cluster.cc:1428] Setting key 12ddc841a7564156a19dbf9d72d93cc9
I20260924 11:52:53.444929 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718: Bootstrap starting.
I20260924 11:52:53.447794 23956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:53.448767 23956 log.cc:824] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:52:53 dist-test-slave-dd32 krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:53.451943 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718: No bootstrap required, opened a new log
I20260924 11:52:53.459554 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:53.439334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53475 (local address 127.1.210.62:42089)
0924 11:52:53.440034 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:53.440046 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:53.440084 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:52:53.441741 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:53.441762 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:53.442149 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:53.442541 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:53.443978 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:53.445384 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:53.446630 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:53.447279 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:53.447466 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:53.452241 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:53.452249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:53.452263 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:53.452300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:53.455438 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:53.456262 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:53.456268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:53.456273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:53.456442 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:53.456828 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:53.456831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:53.456833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:53.457419 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:53.457653 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:53.458117 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 11:52:53.458512 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":310,"threads_started":1}
I20260924 11:52:53.460673 23956 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42089 } }
I20260924 11:52:53.461177 23956 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:53.461290 23956 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15ff2cdedb44c2ea59a403a70ca9718, State: Initialized, Role: FOLLOWER
I20260924 11:52:53.461925 23956 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42089 } }
I20260924 11:52:53.462234 23956 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:53.462374 23956 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:53.462577 23956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:53.463507 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:53.465849 23956 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42089 } }
I20260924 11:52:53.466427 23956 leader_election.cc:302] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15ff2cdedb44c2ea59a403a70ca9718; no voters:
I20260924 11:52:53.467118 23956 leader_election.cc:288] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:53.467235 23962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:53.468763 23962 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [term 1 LEADER]: Becoming Leader. State: Replica: e15ff2cdedb44c2ea59a403a70ca9718, State: Running, Role: LEADER
I20260924 11:52:53.469437 23962 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42089 } }
I20260924 11:52:53.469691 23956 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:53.473078 23965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42089 } } }
I20260924 11:52:53.473531 23965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:53.473050 23967 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15ff2cdedb44c2ea59a403a70ca9718. Latest consensus state: current_term: 1 leader_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15ff2cdedb44c2ea59a403a70ca9718" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 42089 } } }
I20260924 11:52:53.473918 23967 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:53.474335 23973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:53.489003 23973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:53.493664 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:53.470200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55769 (local address 127.1.210.62:42089)
0924 11:52:53.470363 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:53.470367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:53.470901 (+ 534us) server_negotiation.cc:407] Connection header received
0924 11:52:53.471085 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:53.471093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:53.471195 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:53.471325 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:53.475449 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:53.476684 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:53.479162 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:53.480082 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:53.480228 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:53.483757 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:53.483767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:53.483775 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:53.483829 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:53.488764 (+ 4935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:53.489920 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:53.489929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:53.489935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:53.490158 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:53.490725 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:53.490733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:53.490738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:53.491213 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:53.491538 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:53.493036 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0924 11:52:53.493313 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:52:53.499042 23973 catalog_manager.cc:1347] Generated new cluster ID: 60a4ae67b75a43259a5483fd9e1538f3
I20260924 11:52:53.499158 23973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:53.519397 23973 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:53.520797 23973 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:53.528498 23973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15ff2cdedb44c2ea59a403a70ca9718: Generated new TSK 0
I20260924 11:52:53.529338 23973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:53.541466 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 23889
I20260924 11:52:53.542646 23951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:52:53.633984 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 23889
2026-09-24T11:52:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:53 dist-test-slave-dd32 krb5kdc[23988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:53 dist-test-slave-dd32 krb5kdc[23988](info): set up 2 sockets
Sep 24 11:52:53 dist-test-slave-dd32 krb5kdc[23988](info): commencing operation
krb5kdc: starting...
W20260924 11:52:54.749965 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:54 dist-test-slave-dd32 krb5kdc[23988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:55.014429 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41593
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:41593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:55.341125 24001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:55.341461 24001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:55.341626 24001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:55.348814 24001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:55.348966 24001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:55.349093 24001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:55.349288 24001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:55.349342 24001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:55.359282 24001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:41593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:41593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:55.360585 24001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:55.361958 24001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:55.373811 24006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:55.373775 24009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:55.374274 24007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:55.374605 24001 server_base.cc:1034] running on GCE node
I20260924 11:52:55.375366 24001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:55.376650 24001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:55.377871 24001 hybrid_clock.cc:648] HybridClock initialized: now 1790250775377834 us; error 35 us; skew 500 ppm
I20260924 11:52:55.378219 24001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:55 dist-test-slave-dd32 krb5kdc[23988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:55.384876 24001 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:55.386435 24001 webserver.cc:460] Webserver started at http://127.1.210.62:42753/ using document root <none> and password file <none>
I20260924 11:52:55.387102 24001 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:55.387248 24001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:55.387552 24001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:55.390483 24001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "4b17e9cf4ce24d7aa9f9a6475505324a"
format_stamp: "Formatted at 2026-09-24 11:52:55 on dist-test-slave-dd32"
server_key: "d6e06cd3fb732f43ce4ff19a4c9e8cdc"
server_key_iv: "a9799fd2b9bedfb9f4a2e772cfc8059a"
server_key_version: "encryptionkey@0"
I20260924 11:52:55.391223 24001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "4b17e9cf4ce24d7aa9f9a6475505324a"
format_stamp: "Formatted at 2026-09-24 11:52:55 on dist-test-slave-dd32"
server_key: "d6e06cd3fb732f43ce4ff19a4c9e8cdc"
server_key_iv: "a9799fd2b9bedfb9f4a2e772cfc8059a"
server_key_version: "encryptionkey@0"
I20260924 11:52:55.396178 24001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 11:52:55.399621 24016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:55.400945 24001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:52:55.401161 24001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "4b17e9cf4ce24d7aa9f9a6475505324a"
format_stamp: "Formatted at 2026-09-24 11:52:55 on dist-test-slave-dd32"
server_key: "d6e06cd3fb732f43ce4ff19a4c9e8cdc"
server_key_iv: "a9799fd2b9bedfb9f4a2e772cfc8059a"
server_key_version: "encryptionkey@0"
I20260924 11:52:55.401362 24001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:55.417608 24001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:55.421672 24001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:55.421967 24001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:55.440320 24001 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:41593
I20260924 11:52:55.440356 24067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:41593 every 8 connection(s)
I20260924 11:52:55.441713 24001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:55.445407 24068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:55.448110 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24001
I20260924 11:52:55.448280 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:55.448678 1864 external_mini_cluster.cc:1428] Setting key fcca46f9d1590569e465dbb066b4a6f6
I20260924 11:52:55.456904 24068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a: Bootstrap starting.
I20260924 11:52:55.460557 24068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:55.461771 24068 log.cc:824] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:55.462092 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:55.450531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42961 (local address 127.1.210.62:41593)
0924 11:52:55.451305 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:55.451315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:55.451354 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:52:55.453012 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:55.453032 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:55.453493 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:55.453812 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:55.454411 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:55.455742 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:55.457592 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:55.458248 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:55.458381 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:55.459712 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:55.459720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:55.459733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:55.460255 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:55.460324 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:55.460348 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:55.460436 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:55.460803 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 11:52:55.461044 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":360,"threads_started":1}
I20260924 11:52:55.464527 24068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a: No bootstrap required, opened a new log
I20260924 11:52:55.465680 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:55.473357 24068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41593 } }
I20260924 11:52:55.473834 24068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:55.473977 24068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b17e9cf4ce24d7aa9f9a6475505324a, State: Initialized, Role: FOLLOWER
I20260924 11:52:55.474604 24068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41593 } }
I20260924 11:52:55.474907 24068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:55.475057 24068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:55.475235 24068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:55.476643 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:55.469867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38583 (local address 127.1.210.62:41593)
0924 11:52:55.470080 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:55.470089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:55.470519 (+ 430us) server_negotiation.cc:407] Connection header received
0924 11:52:55.470690 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:55.470694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:55.470749 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:55.470828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:55.471445 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:55.472178 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:55.474012 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:55.474601 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:55.474695 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:55.475914 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:55.475918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:55.475922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:55.475939 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:55.475981 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:55.475997 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:55.476078 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:55.476230 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 11:52:55.476299 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:52:55.478101 24068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41593 } }
I20260924 11:52:55.478610 24068 leader_election.cc:302] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b17e9cf4ce24d7aa9f9a6475505324a; no voters:
I20260924 11:52:55.479158 24068 leader_election.cc:288] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:55.479310 24078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:55.479954 24078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [term 1 LEADER]: Becoming Leader. State: Replica: 4b17e9cf4ce24d7aa9f9a6475505324a, State: Running, Role: LEADER
I20260924 11:52:55.480553 24078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41593 } }
I20260924 11:52:55.482416 24078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b17e9cf4ce24d7aa9f9a6475505324a. Latest consensus state: current_term: 1 leader_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41593 } } }
I20260924 11:52:55.482733 24078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:55.483315 24080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b17e9cf4ce24d7aa9f9a6475505324a" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 41593 } } }
I20260924 11:52:55.483729 24080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:55.485168 24068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:55.487394 24082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:55.493000 24082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:55.502074 24082 catalog_manager.cc:1347] Generated new cluster ID: 266594ff0553430fa042d12ac3b6081a
I20260924 11:52:55.502202 24082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:55.517795 24082 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:55.518983 24082 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:55.529007 24082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b17e9cf4ce24d7aa9f9a6475505324a: Generated new TSK 0
I20260924 11:52:55.529911 24082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:55.542158 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24001
W20260924 11:52:55.543188 24063 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42961
W20260924 11:52:55.543921 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:55.544042 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24001
2026-09-24T11:52:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:55 dist-test-slave-dd32 krb5kdc[24099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:55 dist-test-slave-dd32 krb5kdc[24099](info): set up 2 sockets
Sep 24 11:52:55 dist-test-slave-dd32 krb5kdc[24099](info): commencing operation
krb5kdc: starting...
W20260924 11:52:56.641638 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:56 dist-test-slave-dd32 krb5kdc[24099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:56.896327 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36111
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:57.223049 24112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:57.223376 24112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:57.223551 24112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:57.230702 24112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:57.230842 24112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:57.230957 24112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:57.231060 24112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:57.231146 24112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:57.240763 24112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:36111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:57.242206 24112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:57.243503 24112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:57.255148 24118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:57.255167 24117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:57.255290 24120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:57.255916 24112 server_base.cc:1034] running on GCE node
I20260924 11:52:57.256729 24112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:57.258064 24112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:57.259265 24112 hybrid_clock.cc:648] HybridClock initialized: now 1790250777259216 us; error 40 us; skew 500 ppm
I20260924 11:52:57.259562 24112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:57 dist-test-slave-dd32 krb5kdc[24099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250777, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:57.262502 24112 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:57.263861 24112 webserver.cc:460] Webserver started at http://127.1.210.62:42927/ using document root <none> and password file <none>
I20260924 11:52:57.264468 24112 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:57.264567 24112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:57.264823 24112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:57.267688 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "dd3662b43c8046c2b6d6b607cf551733"
format_stamp: "Formatted at 2026-09-24 11:52:57 on dist-test-slave-dd32"
server_key: "96a9fc84dc1c3537be1dd389e0f77ac1"
server_key_iv: "24d5959d2a3039343bc3b33f9c7a9ce9"
server_key_version: "encryptionkey@0"
I20260924 11:52:57.268375 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "dd3662b43c8046c2b6d6b607cf551733"
format_stamp: "Formatted at 2026-09-24 11:52:57 on dist-test-slave-dd32"
server_key: "96a9fc84dc1c3537be1dd389e0f77ac1"
server_key_iv: "24d5959d2a3039343bc3b33f9c7a9ce9"
server_key_version: "encryptionkey@0"
I20260924 11:52:57.273309 24112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:52:57.276618 24127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:57.277956 24112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:57.278134 24112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "dd3662b43c8046c2b6d6b607cf551733"
format_stamp: "Formatted at 2026-09-24 11:52:57 on dist-test-slave-dd32"
server_key: "96a9fc84dc1c3537be1dd389e0f77ac1"
server_key_iv: "24d5959d2a3039343bc3b33f9c7a9ce9"
server_key_version: "encryptionkey@0"
I20260924 11:52:57.278319 24112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:57.283844 24112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:57.287657 24112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:57.287937 24112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:57.305548 24112 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:36111
I20260924 11:52:57.305630 24178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:36111 every 8 connection(s)
I20260924 11:52:57.306905 24112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:57.310050 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24112
I20260924 11:52:57.310216 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:57.310619 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:57.310669 1864 external_mini_cluster.cc:1428] Setting key bc83d6aef6361f1d9437f9a3cadd50eb
I20260924 11:52:57.318415 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:57.312312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38091 (local address 127.1.210.62:36111)
0924 11:52:57.313154 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:57.313166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:57.313208 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:52:57.315095 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:57.315116 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:57.315441 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:57.315760 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:57.316838 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:57.316845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:57.316857 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:57.316871 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:57.316936 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:57.316955 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:57.317196 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:57.317300 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 11:52:57.317519 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":364,"threads_started":1}
I20260924 11:52:57.322099 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:52:57.324419 24179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733: Bootstrap starting.
I20260924 11:52:57.328305 24179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733: Neither blocks nor log segments found. Creating new log.
I20260924 11:52:57.329555 24179 log.cc:824] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733: Log is configured to *not* fsync() on all Append() calls
I20260924 11:52:57.331696 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:57.328962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56733 (local address 127.1.210.62:36111)
0924 11:52:57.329185 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:57.329194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:57.329721 (+ 527us) server_negotiation.cc:407] Connection header received
0924 11:52:57.329895 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:57.329901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:57.329965 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:57.330112 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:57.331102 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:57.331108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:57.331112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:52:57.331128 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:57.331172 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:52:57.331190 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:57.331306 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:57.331376 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 11:52:57.331433 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:52:57.333000 24179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733: No bootstrap required, opened a new log
I20260924 11:52:57.340150 24179 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3662b43c8046c2b6d6b607cf551733" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36111 } }
I20260924 11:52:57.340531 24179 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:52:57.340654 24179 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3662b43c8046c2b6d6b607cf551733, State: Initialized, Role: FOLLOWER
I20260924 11:52:57.341266 24179 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3662b43c8046c2b6d6b607cf551733" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36111 } }
I20260924 11:52:57.341558 24179 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:52:57.341707 24179 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:52:57.341872 24179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:52:57.344158 24179 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3662b43c8046c2b6d6b607cf551733" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36111 } }
I20260924 11:52:57.344653 24179 leader_election.cc:302] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd3662b43c8046c2b6d6b607cf551733; no voters:
I20260924 11:52:57.345280 24179 leader_election.cc:288] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:52:57.345428 24189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:52:57.346225 24189 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [term 1 LEADER]: Becoming Leader. State: Replica: dd3662b43c8046c2b6d6b607cf551733, State: Running, Role: LEADER
I20260924 11:52:57.346732 24189 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3662b43c8046c2b6d6b607cf551733" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36111 } }
I20260924 11:52:57.347385 24179 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:52:57.349732 24191 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd3662b43c8046c2b6d6b607cf551733. Latest consensus state: current_term: 1 leader_uuid: "dd3662b43c8046c2b6d6b607cf551733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3662b43c8046c2b6d6b607cf551733" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36111 } } }
I20260924 11:52:57.350173 24191 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:57.349851 24190 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd3662b43c8046c2b6d6b607cf551733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3662b43c8046c2b6d6b607cf551733" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 36111 } } }
I20260924 11:52:57.350441 24190 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733 [sys.catalog]: This master's current role is: LEADER
I20260924 11:52:57.352751 24196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:52:57.357849 24196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:52:57.365928 24196 catalog_manager.cc:1347] Generated new cluster ID: fdaea6419a324af6966df491b6114226
I20260924 11:52:57.366042 24196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:52:57.377777 24196 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:52:57.378227 24196 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:52:57.385973 24196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd3662b43c8046c2b6d6b607cf551733: Generated new TSK 0
I20260924 11:52:57.386852 24196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:52:57.395496 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24112
W20260924 11:52:57.396540 24174 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38091
W20260924 11:52:57.397245 1864 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24112: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:52:57.397380 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24112
2026-09-24T11:52:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1852 ms)
[----------] 18 tests from AuthTokenIssuingTest (34668 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:52:57 dist-test-slave-dd32 krb5kdc[24210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:52:57 dist-test-slave-dd32 krb5kdc[24210](info): set up 2 sockets
Sep 24 11:52:57 dist-test-slave-dd32 krb5kdc[24210](info): commencing operation
krb5kdc: starting...
W20260924 11:52:58.497702 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:52:58 dist-test-slave-dd32 krb5kdc[24210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:52:58.747898 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:46301
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:46301,127.1.210.61:43779,127.1.210.60:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:52:59.073959 24223 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:52:59.074337 24223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:59.074479 24223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:52:59.074633 24223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:52:59.081956 24223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:59.082052 24223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:59.082150 24223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:59.082227 24223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:59.082345 24223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:59.092250 24223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:46301,127.1.210.61:43779,127.1.210.60:45787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:59.093603 24223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:59.094907 24223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:59.106431 24228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:59.106524 24229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:59.106491 24231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:59.107443 24223 server_base.cc:1034] running on GCE node
I20260924 11:52:59.108145 24223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:59.109468 24223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:59.110679 24223 hybrid_clock.cc:648] HybridClock initialized: now 1790250779110602 us; error 49 us; skew 500 ppm
I20260924 11:52:59.110960 24223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:59 dist-test-slave-dd32 krb5kdc[24210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:59.114106 24223 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:52:59.115437 24223 webserver.cc:460] Webserver started at http://127.1.210.62:35043/ using document root <none> and password file <none>
I20260924 11:52:59.116106 24223 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:59.116220 24223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:59.116541 24223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:59.119478 24223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "9632901ad0d644549605ffa7f719fc67"
format_stamp: "Formatted at 2026-09-24 11:52:59 on dist-test-slave-dd32"
server_key: "23f557a86e28ce564c88734c5b758693"
server_key_iv: "06134bd73f904e4bf8293e31fe30b5eb"
server_key_version: "encryptionkey@0"
I20260924 11:52:59.120241 24223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "9632901ad0d644549605ffa7f719fc67"
format_stamp: "Formatted at 2026-09-24 11:52:59 on dist-test-slave-dd32"
server_key: "23f557a86e28ce564c88734c5b758693"
server_key_iv: "06134bd73f904e4bf8293e31fe30b5eb"
server_key_version: "encryptionkey@0"
I20260924 11:52:59.125308 24223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:52:59.128482 24238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:59.129762 24223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:59.129941 24223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "9632901ad0d644549605ffa7f719fc67"
format_stamp: "Formatted at 2026-09-24 11:52:59 on dist-test-slave-dd32"
server_key: "23f557a86e28ce564c88734c5b758693"
server_key_iv: "06134bd73f904e4bf8293e31fe30b5eb"
server_key_version: "encryptionkey@0"
I20260924 11:52:59.130144 24223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:59.148693 24223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:59.152735 24223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:59.153009 24223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:59.170955 24223 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:46301
I20260924 11:52:59.170971 24289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:46301 every 8 connection(s)
I20260924 11:52:59.172271 24223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:52:59.175958 24290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:59.181682 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24223
I20260924 11:52:59.181890 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:52:59.182508 1864 external_mini_cluster.cc:1428] Setting key 09df7d824402e47c66a25966715facb9
I20260924 11:52:59.182397 24290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:52:59 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:52:59.207515 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.185497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:46301 (local address 127.0.0.1:48480)
0924 11:52:59.186678 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.186728 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:59.189439 (+ 2711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:59.190301 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:59.190311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:59.191144 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:59.191657 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.191683 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.193570 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.193574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.194624 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.194636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.195939 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.195944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.196069 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.196731 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:59.196753 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:59.199169 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:59.203526 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.203536 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.203544 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.203920 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.204384 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.204387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.204389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.204535 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.205587 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:59.205593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:59.205822 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 11:52:59.206037 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 11:52:59.206451 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"spinlock_wait_cycles":33152,"thread_start_us":704,"threads_started":1}
I20260924 11:52:59.207553 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.185505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.1.210.62:46301)
0924 11:52:59.186375 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:59.186394 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:59.186865 (+ 471us) server_negotiation.cc:407] Connection header received
0924 11:52:59.189820 (+ 2955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:59.189825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:59.190049 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:59.190209 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:59.192013 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.193415 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.194766 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.195730 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.195913 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.199391 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:59.199403 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:59.199422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:59.199469 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:59.203310 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.204057 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.204068 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.204075 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.204271 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.204659 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.204667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.204673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.205439 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:59.205672 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:59.206240 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 11:52:59.206680 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":606,"thread_start_us":362,"threads_started":1}
WARNING: no policy specified for kudu/127.1.210.61@KRBTEST.COM; defaulting to no policy
I20260924 11:52:59.210722 24290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:52:59.211781 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.211381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:42942)
0924 11:52:59.211551 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.211670 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.61@KRBTEST.COM" created.
W20260924 11:52:59.212139 24240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
I20260924 11:52:59.213773 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.213428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:42958)
0924 11:52:59.213576 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.213651 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:52:59.214013 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:43779: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab.
Entry for principal kudu/127.1.210.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab.
I20260924 11:52:59.243903 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:43779
--webserver_interface=127.1.210.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab
--principal=kudu/127.1.210.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:46301,127.1.210.61:43779,127.1.210.60:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:52:59.263371 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } attempt: 1
I20260924 11:52:59.264719 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.264275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:42970)
0924 11:52:59.264471 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.264561 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:52:59.265743 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.265370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:42986)
0924 11:52:59.265563 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.265630 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:52:59.265925 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:43779: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
I20260924 11:52:59.334146 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } attempt: 2
I20260924 11:52:59.335492 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.334928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:42998)
0924 11:52:59.335174 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.335287 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 11:52:59.336504 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.336128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43006)
0924 11:52:59.336274 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.336370 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 11:52:59.336699 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:43779: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
I20260924 11:52:59.427974 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } attempt: 3
I20260924 11:52:59.429131 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.428635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43018)
0924 11:52:59.428835 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.428927 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:52:59.430079 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.429755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43032)
0924 11:52:59.429891 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.429962 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:52:59.430234 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:43779: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
I20260924 11:52:59.573442 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } attempt: 4
W20260924 11:52:59.574138 24297 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:52:59.574581 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:52:59.574695 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260924 11:52:59.574695 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.574070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43034)
0924 11:52:59.574400 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.574508 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260924 11:52:59.574826 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:52:59.575784 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.575380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43042)
0924 11:52:59.575539 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.575637 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:52:59.575996 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:43779: Network error: Client connection negotiation failed: client connection to 127.1.210.61:43779: connect: Connection refused (error 111)
W20260924 11:52:59.582597 24297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:52:59.582679 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:52:59.582825 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:52:59.582899 24297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:52:59.583005 24297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:52:59.592923 24297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:46301,127.1.210.61:43779,127.1.210.60:45787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab
--principal=kudu/127.1.210.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:43779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:52:59.594266 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:52:59.595578 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:52:59.607438 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:59.607396 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:52:59.607489 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:52:59.610531 24297 server_base.cc:1034] running on GCE node
I20260924 11:52:59.611277 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:52:59.612543 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:52:59.613761 24297 hybrid_clock.cc:648] HybridClock initialized: now 1790250779613683 us; error 51 us; skew 500 ppm
I20260924 11:52:59.614094 24297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:52:59 dist-test-slave-dd32 krb5kdc[24210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:52:59.617370 24297 init.cc:377] Logged in from keytab as kudu/127.1.210.61@KRBTEST.COM (short username kudu)
I20260924 11:52:59.618860 24297 webserver.cc:460] Webserver started at http://127.1.210.61:41559/ using document root <none> and password file <none>
I20260924 11:52:59.619578 24297 fs_manager.cc:359] Metadata directory not provided
I20260924 11:52:59.619694 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:52:59.620023 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:52:59.622974 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data/instance:
uuid: "438f414d5e1d4989a7801a6bed270ab7"
format_stamp: "Formatted at 2026-09-24 11:52:59 on dist-test-slave-dd32"
server_key: "5d633ed53681e85101f616bf4ccc25d4"
server_key_iv: "a06b8d56120c99ec726108bd72361469"
server_key_version: "encryptionkey@0"
I20260924 11:52:59.623755 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal/instance:
uuid: "438f414d5e1d4989a7801a6bed270ab7"
format_stamp: "Formatted at 2026-09-24 11:52:59 on dist-test-slave-dd32"
server_key: "5d633ed53681e85101f616bf4ccc25d4"
server_key_iv: "a06b8d56120c99ec726108bd72361469"
server_key_version: "encryptionkey@0"
I20260924 11:52:59.628685 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 11:52:59.631942 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:52:59.633297 24297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:52:59.633462 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal
uuid: "438f414d5e1d4989a7801a6bed270ab7"
format_stamp: "Formatted at 2026-09-24 11:52:59 on dist-test-slave-dd32"
server_key: "5d633ed53681e85101f616bf4ccc25d4"
server_key_iv: "a06b8d56120c99ec726108bd72361469"
server_key_version: "encryptionkey@0"
I20260924 11:52:59.633664 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:52:59.648495 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:52:59.652714 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:52:59.652983 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:52:59.671098 24297 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.61:43779
I20260924 11:52:59.671118 24363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.61:43779 every 8 connection(s)
I20260924 11:52:59.672377 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
I20260924 11:52:59.675850 24364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:52:59.678023 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24297
I20260924 11:52:59.678221 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-1/wal/instance
I20260924 11:52:59.678637 1864 external_mini_cluster.cc:1428] Setting key 774914ff1cabc27b2bdc3c9566e60ffe
I20260924 11:52:59.682286 24364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:52:59 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.60@KRBTEST.COM; defaulting to no policy
I20260924 11:52:59.704532 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.684811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.1.210.62:46301)
0924 11:52:59.684985 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:59.684989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:59.686369 (+ 1380us) server_negotiation.cc:407] Connection header received
0924 11:52:59.688643 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:59.688653 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:59.688751 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:59.688870 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:59.690326 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.691471 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.692930 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.693715 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.693863 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.697487 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:59.697497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:59.697505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:59.697553 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:59.701378 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.702265 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.702271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.702276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.702430 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.702853 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.702861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.702866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.703249 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:59.703425 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:59.704088 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 11:52:59.704256 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.60@KRBTEST.COM" created.
I20260924 11:52:59.705775 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.685401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:46301 (local address 127.0.0.1:48490)
0924 11:52:59.686158 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.686222 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:59.688352 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:59.689098 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:59.689113 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:59.689604 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:59.690144 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.690160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.691615 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.691619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.692775 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.692786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.693915 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.693920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.694123 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.694820 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:59.694841 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:59.697249 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:59.701546 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.701556 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.701576 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.702121 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.702552 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.702556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.702558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.702717 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.703403 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:59.703413 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:59.703735 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 11:52:59.703955 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 11:52:59.704422 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":341,"threads_started":1}
I20260924 11:52:59.707679 24364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:52:59 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:52:59.723867 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.708474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43052)
0924 11:52:59.708699 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.708722 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:59.708826 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:59.709759 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:59.709785 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:59.710284 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:59.710486 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.710500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.711887 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.711895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.713428 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.713445 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.714276 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.714288 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.714479 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.715092 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:59.715108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:59.717349 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:59.721626 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.721629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.721632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.721955 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.722367 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.722370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.722372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.722499 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.723221 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:59.723224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:59.723329 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 11:52:59.723434 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 11:52:59.723617 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:52:59.724581 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.708638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.1.210.61:43779)
0924 11:52:59.709350 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:59.709356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:59.709380 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:52:59.709424 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:59.709429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:59.709600 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:59.709742 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:59.710672 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.711743 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.713582 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.714117 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.714211 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.717559 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:59.717574 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:59.717595 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:59.717638 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:59.721504 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.722095 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.722106 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.722113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.722259 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.722610 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.722618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.722623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.723098 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:59.723316 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:59.723624 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 11:52:59.723903 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":392,"threads_started":1}
I20260924 11:52:59.727216 24364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:52:59.728276 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.727853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52798)
0924 11:52:59.728020 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.728146 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 11:52:59.728600 24314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
I20260924 11:52:59.730171 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.729871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52800)
0924 11:52:59.730006 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.730085 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:52:59.730485 24364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab.
Entry for principal kudu/127.1.210.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab.
I20260924 11:52:59.740190 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:45787
--webserver_interface=127.1.210.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab
--principal=kudu/127.1.210.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:46301,127.1.210.61:43779,127.1.210.60:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:52:59.779788 24364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 1
I20260924 11:52:59.780956 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.780521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52802)
0924 11:52:59.780706 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.780808 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 11:52:59.782058 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.781743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52810)
0924 11:52:59.781882 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.781951 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:52:59.782361 24364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
I20260924 11:52:59.850593 24364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 2
I20260924 11:52:59.851794 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.851268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52818)
0924 11:52:59.851459 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.851571 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:52:59.852955 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.852570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52832)
0924 11:52:59.852715 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.852799 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:52:59.853235 24364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
I20260924 11:52:59.875278 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } attempt: 5
Sep 24 11:52:59 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:52:59.888851 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.876381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.1.210.61:43779)
0924 11:52:59.876612 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:52:59.876618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:52:59.876643 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:52:59.876695 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:52:59.876698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:52:59.876772 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:52:59.876882 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:52:59.877752 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.878520 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.879695 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.880141 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.880279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.883665 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:52:59.883670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:52:59.883673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:52:59.883705 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:52:59.886267 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.886927 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.886939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.886942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.887068 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:52:59.887444 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:52:59.887450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:52:59.887453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:52:59.887730 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:52:59.887880 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:52:59.888282 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 11:52:59.888493 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:52:59.888851 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.876126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43058)
0924 11:52:59.876315 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.876342 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:52:59.876473 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:52:59.876925 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:52:59.876931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:52:59.877413 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:52:59.877597 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.877607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.878666 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.878672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.879534 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:52:59.879554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:52:59.880303 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:52:59.880310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:52:59.880528 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:52:59.881221 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:52:59.881239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:52:59.883457 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:52:59.886412 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.886417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.886421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.886802 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.887182 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:52:59.887188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:52:59.887191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:52:59.887337 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:52:59.887859 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:52:59.887863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:52:59.888019 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:52:59.888187 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 11:52:59.888472 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:52:59.890167 24290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:52:59.891152 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.890781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52844)
0924 11:52:59.890942 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.891019 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:52:59.892092 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.891767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52846)
0924 11:52:59.891910 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.891993 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:52:59.892306 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
I20260924 11:52:59.943599 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 1
I20260924 11:52:59.944456 24364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 3
I20260924 11:52:59.944941 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.944265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52860)
0924 11:52:59.944544 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.944692 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 11:52:59.945604 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.944973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52864)
0924 11:52:59.945313 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.945413 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260924 11:52:59.946321 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.945759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52874)
0924 11:52:59.946037 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.946151 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260924 11:52:59.946539 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
I20260924 11:52:59.946627 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:52:59.946160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52878)
0924 11:52:59.946376 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:52:59.946485 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260924 11:52:59.946839 24364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
I20260924 11:53:00.014860 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 2
I20260924 11:53:00.016024 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.015573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52890)
0924 11:53:00.015781 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.015876 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:53:00.016987 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.016666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52904)
0924 11:53:00.016813 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.016888 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:53:00.017220 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
W20260924 11:53:00.085702 24371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:53:00.086095 24371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:53:00.086226 24371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:53:00.086367 24371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:53:00.090040 24364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 4
I20260924 11:53:00.091162 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.090676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52916)
0924 11:53:00.090909 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.090993 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:53:00.092115 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.091806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52924)
0924 11:53:00.091944 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.092014 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:53:00.092368 24364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
W20260924 11:53:00.093916 24371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:53:00.093994 24371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:53:00.094141 24371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:53:00.094216 24371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:53:00.094301 24371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:53:00.104122 24371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:46301,127.1.210.61:43779,127.1.210.60:45787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab
--principal=kudu/127.1.210.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:53:00.105453 24371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:53:00.106783 24371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:53:00.118734 24376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:00.118852 24379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:00.118700 24377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:53:00.119416 24371 server_base.cc:1034] running on GCE node
I20260924 11:53:00.120189 24371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:53:00.121572 24371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:53:00.122822 24371 hybrid_clock.cc:648] HybridClock initialized: now 1790250780122774 us; error 51 us; skew 500 ppm
I20260924 11:53:00.123140 24371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:53:00.123402 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 3
I20260924 11:53:00.124387 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.123941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52930)
0924 11:53:00.124153 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.124241 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 11:53:00.125473 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.125042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52944)
0924 11:53:00.125245 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.125335 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 11:53:00.125694 24290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:45787: Network error: Client connection negotiation failed: client connection to 127.1.210.60:45787: connect: Connection refused (error 111)
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250780, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:53:00.126683 24371 init.cc:377] Logged in from keytab as kudu/127.1.210.60@KRBTEST.COM (short username kudu)
I20260924 11:53:00.128086 24371 webserver.cc:460] Webserver started at http://127.1.210.60:41977/ using document root <none> and password file <none>
I20260924 11:53:00.128751 24371 fs_manager.cc:359] Metadata directory not provided
I20260924 11:53:00.128878 24371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:53:00.129237 24371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:53:00.132040 24371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data/instance:
uuid: "9d9df49427bf49da8364c83289c207de"
format_stamp: "Formatted at 2026-09-24 11:53:00 on dist-test-slave-dd32"
server_key: "b2a358e9ba936500282a3625f5738750"
server_key_iv: "a174666bdc9c58668efde720faf2d21f"
server_key_version: "encryptionkey@0"
I20260924 11:53:00.132798 24371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal/instance:
uuid: "9d9df49427bf49da8364c83289c207de"
format_stamp: "Formatted at 2026-09-24 11:53:00 on dist-test-slave-dd32"
server_key: "b2a358e9ba936500282a3625f5738750"
server_key_iv: "a174666bdc9c58668efde720faf2d21f"
server_key_version: "encryptionkey@0"
I20260924 11:53:00.138123 24371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 11:53:00.141450 24386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:53:00.142640 24371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:53:00.142808 24371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal
uuid: "9d9df49427bf49da8364c83289c207de"
format_stamp: "Formatted at 2026-09-24 11:53:00 on dist-test-slave-dd32"
server_key: "b2a358e9ba936500282a3625f5738750"
server_key_iv: "a174666bdc9c58668efde720faf2d21f"
server_key_version: "encryptionkey@0"
I20260924 11:53:00.143019 24371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:53:00.168525 24371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:53:00.173005 24371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:53:00.173307 24371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:53:00.191469 24371 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.60:45787
I20260924 11:53:00.191476 24437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.60:45787 every 8 connection(s)
I20260924 11:53:00.192816 24371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
I20260924 11:53:00.194051 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24371
I20260924 11:53:00.194229 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790250429340952-1864-0/minicluster-data/master-2/wal/instance
I20260924 11:53:00.194749 1864 external_mini_cluster.cc:1428] Setting key 988972c390b94f2a02001c0fdf59ad7a
I20260924 11:53:00.197371 24438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:53:00.205647 24438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250780, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:53:00.217582 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.196544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.1.210.60:45787)
0924 11:53:00.197319 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.197330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.197374 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:53:00.198978 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.198998 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.199353 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.199680 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.201414 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.202721 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.204491 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.205346 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.205478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.210435 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.210443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.210452 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.210490 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.213376 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.214046 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.214058 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.214063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.214215 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.214598 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.214605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.214608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.215108 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.215433 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.215852 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.216326 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":353,"threads_started":1}
I20260924 11:53:00.223862 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.208580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:46301 (local address 127.0.0.1:48492)
0924 11:53:00.209436 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.209477 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:00.209606 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:00.210249 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:00.210255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:00.210913 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:00.211315 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.211326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.212591 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.212596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.213738 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.213749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.214473 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.214479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.214655 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.215252 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:00.215268 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:00.217530 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:00.221307 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.221319 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.221330 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.221727 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.222124 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.222129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.222133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.222271 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.222801 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:00.222808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:00.222977 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 11:53:00.223142 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 11:53:00.223455 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":382,"threads_started":1}
I20260924 11:53:00.223913 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.208642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.1.210.62:46301)
0924 11:53:00.209940 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.209945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.209959 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:53:00.210007 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.210011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.210069 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.210193 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.211469 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.212428 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.213883 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.214294 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.214440 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.217729 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.217733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.217737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.217769 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.221169 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.221864 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.221870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.221874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.222010 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.222386 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.222390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.222394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.222666 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.222827 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.223270 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.223492 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":1100,"threads_started":1}
I20260924 11:53:00.225405 24438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250780, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:53:00.236219 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.220729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.1.210.62:46301)
0924 11:53:00.221372 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.221376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.221395 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:53:00.221433 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.221436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.221487 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.221628 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.222808 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.223618 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.225267 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.225732 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.225864 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.230476 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.230483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.230487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.230523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.233500 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.234134 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.234138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.234143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.234297 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.234734 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.234739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.234742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.235100 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.235367 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.235579 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.235787 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":328,"threads_started":1}
I20260924 11:53:00.239756 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.226127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:43779 (local address 127.0.0.1:43066)
0924 11:53:00.226274 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.226293 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:00.226383 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:00.226690 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:00.226694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:00.227012 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:00.227143 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.227150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.228546 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.228549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.229525 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.229535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.230469 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.230474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.230677 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.231167 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:00.231190 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:00.233528 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:00.237390 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.237393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.237396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.237917 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.238300 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.238306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.238309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.238488 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.239040 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:00.239045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:00.239220 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 11:53:00.239367 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 11:53:00.239544 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:53:00.240134 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.226139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.1.210.61:43779)
0924 11:53:00.226348 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.226356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.226375 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:53:00.226476 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.226483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.226559 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.226672 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.227289 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.228420 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.229658 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.230302 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.230440 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.233726 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.233731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.233734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.233762 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.237234 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.238057 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.238062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.238064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.238187 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.238618 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.238623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.238626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.238907 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.239070 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.239501 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.239730 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:53:00.241010 24438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250780, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:53:00.253996 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.241614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52958)
0924 11:53:00.241732 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.241749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:00.241860 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:00.242149 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:00.242152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:00.242451 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:00.242572 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.242579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.243718 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.243722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.244663 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.244674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.245693 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.245697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.245838 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.246401 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:00.246416 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:00.248627 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:00.251608 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.251615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.251619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.251950 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.252313 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.252321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.252325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.252445 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.252910 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:00.252914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:00.253048 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 11:53:00.253264 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 11:53:00.253590 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":3}
I20260924 11:53:00.254079 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.241695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.1.210.60:45787)
0924 11:53:00.241874 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.241882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.241902 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:53:00.241945 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.241949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.242016 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.242127 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.242709 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.243597 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.244804 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.245543 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.245684 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.248778 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.248783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.248788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.248826 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.251473 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.252071 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.252076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.252080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.252203 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.252545 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.252550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.252553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.252791 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.252933 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.253345 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.253743 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":53}
I20260924 11:53:00.254124 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.238252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.1.210.61:43779)
0924 11:53:00.238824 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.238828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.238846 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:53:00.238881 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.238884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.238933 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.239054 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.240540 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.241791 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.243287 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.243727 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.243849 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.248701 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.248706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.248709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.248739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.251452 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.252124 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.252131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.252134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.252261 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.252640 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.252646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.252650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.252901 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.253206 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.253461 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.253675 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260924 11:53:00.259212 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:53:00.259943 24438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: Bootstrap starting.
I20260924 11:53:00.263154 24438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: Neither blocks nor log segments found. Creating new log.
I20260924 11:53:00.264189 24438 log.cc:824] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: Log is configured to *not* fsync() on all Append() calls
I20260924 11:53:00.261775 24265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9632901ad0d644549605ffa7f719fc67"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498
I20260924 11:53:00.266961 24438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: No bootstrap required, opened a new log
I20260924 11:53:00.273837 24438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.274179 24438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9df49427bf49da8364c83289c207de, State: Initialized, Role: FOLLOWER
I20260924 11:53:00.274601 24438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.275431 24447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.275763 24447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:00.276742 24438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:53:00.286298 24458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:53:00.286424 24458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:53:00.284091 24371 master_runner.cc:186] Error getting master registration for 127.1.210.62:46301: OK, instance_id { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" instance_seqno: 1790250779165855 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:53:00.292012 24371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:53:00.302924 24290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 4
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:53:00.318691 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.303704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.1.210.60:45787)
0924 11:53:00.303924 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.303930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.304023 (+ 93us) server_negotiation.cc:407] Connection header received
0924 11:53:00.304251 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.304256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.304336 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.304449 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.305320 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.306074 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.307323 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.307786 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.307930 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.312044 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.312053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.312077 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.312120 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.315628 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.316456 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.316461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.316464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.316607 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.317034 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.317039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.317042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.317492 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.317668 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.318087 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.318302 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:53:00.318699 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.303650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52972)
0924 11:53:00.303912 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.303929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:00.304108 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:00.304479 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:00.304485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:00.304944 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:00.305174 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.305186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.306219 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.306226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.307171 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.307183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.307942 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.307948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.308143 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.308808 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:00.308834 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:00.311775 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:00.315786 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.315791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.315796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.316325 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.316731 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.316737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.316742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.316915 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.317627 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:00.317632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:00.317818 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 11:53:00.317979 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:53:00.318289 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 11:53:00.323249 24290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: Bootstrap starting.
I20260924 11:53:00.327306 24290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: Neither blocks nor log segments found. Creating new log.
I20260924 11:53:00.328578 24290 log.cc:824] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: Log is configured to *not* fsync() on all Append() calls
I20260924 11:53:00.331897 24290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: No bootstrap required, opened a new log
I20260924 11:53:00.339268 24290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.339680 24290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9632901ad0d644549605ffa7f719fc67, State: Initialized, Role: FOLLOWER
I20260924 11:53:00.340176 24290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.341441 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.342115 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:00.342694 24290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:53:00.342804 24265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 11:53:00.342978 24265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:53:00.347281 24265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.349920 24265 leader_election.cc:288] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [CANDIDATE]: Term 1 election: Requested vote from peers 438f414d5e1d4989a7801a6bed270ab7 (127.1.210.61:43779), 9d9df49427bf49da8364c83289c207de (127.1.210.60:45787)
I20260924 11:53:00.351102 24413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9632901ad0d644549605ffa7f719fc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9d9df49427bf49da8364c83289c207de"
I20260924 11:53:00.351580 24413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:53:00.354975 24413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9632901ad0d644549605ffa7f719fc67 in term 1.
I20260924 11:53:00.357018 24240 leader_election.cc:302] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9632901ad0d644549605ffa7f719fc67, 9d9df49427bf49da8364c83289c207de; no voters:
I20260924 11:53:00.357729 24461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:53:00.358424 24461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [term 1 LEADER]: Becoming Leader. State: Replica: 9632901ad0d644549605ffa7f719fc67, State: Running, Role: LEADER
I20260924 11:53:00.358980 24461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
W20260924 11:53:00.361788 24472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:53:00.361953 24472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:53:00.367147 24474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9632901ad0d644549605ffa7f719fc67. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.367548 24474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: This master's current role is: LEADER
I20260924 11:53:00.368367 24480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:53:00.370575 24480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:53:00.392158 24364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } attempt: 5
I20260924 11:53:00.393857 24413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [term 1 FOLLOWER]: Refusing update from remote peer 9632901ad0d644549605ffa7f719fc67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:53:00.397512 24474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:53:00.405671 24244 tablet.cc:2369] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:53:00.370368 24223 master_runner.cc:186] Error getting master registration for 127.1.210.61:43779: OK, instance_id { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" instance_seqno: 1790250779665998 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:53:00.408991 24223 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:53:00.423828 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.379746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.1.210.60:45787)
0924 11:53:00.379913 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.379918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.392864 (+ 12946us) server_negotiation.cc:407] Connection header received
0924 11:53:00.393274 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.393279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.393381 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.393453 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.397746 (+ 4293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.398832 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.400707 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.402139 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.402268 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.412547 (+ 10279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.412557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.412581 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.412636 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.417103 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.420778 (+ 3675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.420790 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.420796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.420986 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.421582 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.421590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.421596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.422058 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.422364 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.422999 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.423290 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 11:53:00.425604 24392 tablet.cc:2369] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:53:00.426684 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.375615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.1.210.62:46301)
0924 11:53:00.375804 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.375810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.380509 (+ 4699us) server_negotiation.cc:407] Connection header received
0924 11:53:00.380755 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.380761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.380831 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.380894 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.384950 (+ 4056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.386118 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.394137 (+ 8019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.394785 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.394897 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.410877 (+ 15980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.410885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.410890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.410934 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.414395 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.420699 (+ 6304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.420705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.420709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.420854 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.422619 (+ 1765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.422625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.422629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.422978 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.423216 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.424819 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.426318 (+ 1499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:53:00.427837 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.368830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.1.210.61:43779)
0924 11:53:00.369293 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.369301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.376887 (+ 7586us) server_negotiation.cc:407] Connection header received
0924 11:53:00.377680 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.377687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.377768 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.377840 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.381465 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.382563 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.391163 (+ 8600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.391868 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.391974 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.415319 (+ 23345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.415326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.415331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.415374 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.418958 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.420364 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.420369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.420373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.420518 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.423668 (+ 3150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.423673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.423676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.424027 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.424259 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.427272 (+ 3013us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.427499 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260924 11:53:00.438457 24447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9632901ad0d644549605ffa7f719fc67. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.438792 24447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 11:53:00 dist-test-slave-dd32 krb5kdc[24210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:53:00.443463 24484 mvcc.cc:205] Tried to move back new op lower bound from 7332867196482039808 to 7332867196355547136. Current Snapshot: MvccSnapshot[applied={T|T < 7332867196482039808}]
I20260924 11:53:00.448571 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.392688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:45787 (local address 127.0.0.1:52990)
0924 11:53:00.393009 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:00.393026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:00.393136 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:00.431070 (+ 37934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:00.431075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:00.431586 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:00.431808 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.431821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.434074 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.434080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.435531 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:00.435547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.436811 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.436817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:00.437038 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.437821 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:00.437843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:00.440721 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:00.445378 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.445384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.445388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.445903 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.446390 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:00.446395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:00.446399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:00.446578 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:00.447259 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:00.447266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:00.447436 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 11:53:00.447599 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 11:53:00.447896 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260924 11:53:00.450083 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:00.393424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.1.210.60:45787)
0924 11:53:00.422194 (+ 28770us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.422201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.422223 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:53:00.422279 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.422284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.422364 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.431978 (+ 9614us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.432281 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.433917 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.435869 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:00.436622 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:00.440894 (+ 4272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:00.441169 (+ 275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:00.441176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:00.441181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:00.441213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:00.445218 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.446067 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.446075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.446079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.446265 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:00.446758 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:00.446763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:00.446767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:00.447121 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:00.449338 (+ 2217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:00.449455 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 11:53:00.449692 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28624,"thread_start_us":29005,"threads_started":1}
I20260924 11:53:00.454559 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.454887 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: This master's current role is: LEADER
I20260924 11:53:00.455659 24480 catalog_manager.cc:1347] Generated new cluster ID: 5783a281de454a99be82557e86a22c00
I20260924 11:53:00.455754 24480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:53:00.461851 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: Bootstrap starting.
I20260924 11:53:00.468876 24447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.469202 24447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:00.474546 24364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: Neither blocks nor log segments found. Creating new log.
I20260924 11:53:00.476114 24364 log.cc:824] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:53:00.487177 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: No bootstrap required, opened a new log
I20260924 11:53:00.502651 24364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.503118 24364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438f414d5e1d4989a7801a6bed270ab7, State: Initialized, Role: FOLLOWER
I20260924 11:53:00.503675 24364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } }
I20260924 11:53:00.506479 24480 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:53:00.508303 24480 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:53:00.517593 24491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.518018 24491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:00.519223 24364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:53:00.519872 24338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:53:00.523634 24338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [term 1 FOLLOWER]: Refusing update from remote peer 9632901ad0d644549605ffa7f719fc67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:53:00.524202 24339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9632901ad0d644549605ffa7f719fc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "438f414d5e1d4989a7801a6bed270ab7"
I20260924 11:53:00.524457 24474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:53:00.527787 24339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9632901ad0d644549605ffa7f719fc67 in term 1.
I20260924 11:53:00.531746 24480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67: Generated new TSK 0
I20260924 11:53:00.532800 24480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:53:00.599207 24491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9632901ad0d644549605ffa7f719fc67. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.599548 24491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:00.599910 24502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.600237 24502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:00.606986 24508 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 11:53:00.646344 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.643430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.1.210.61:43779)
0924 11:53:00.643647 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.643655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.644161 (+ 506us) server_negotiation.cc:407] Connection header received
0924 11:53:00.644475 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.644484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.644582 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.644676 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.646075 (+ 1399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43086: BlockingRecv error: recv got EOF from 127.0.0.1:43086 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 11:53:00.649343 24485 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.644546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.1.210.60:45787)
0924 11:53:00.644936 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.644942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.647217 (+ 2275us) server_negotiation.cc:407] Connection header received
0924 11:53:00.647468 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.647473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.647585 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.647671 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.648901 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52992: BlockingRecv error: recv got EOF from 127.0.0.1:52992 (error 108)
Metrics: {"server-negotiator.queue_time_us":253}
W20260924 11:53:00.651940 24444 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.649542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.1.210.62:46301)
0924 11:53:00.649727 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.649733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.650332 (+ 599us) server_negotiation.cc:407] Connection header received
0924 11:53:00.650593 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.650599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.650678 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.650763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.651654 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48518: BlockingRecv error: recv got EOF from 127.0.0.1:48518 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 11:53:00.658563 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.656530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.1.210.61:43779)
0924 11:53:00.656710 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.656716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.657166 (+ 450us) server_negotiation.cc:407] Connection header received
0924 11:53:00.657384 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.657391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.657523 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.657641 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.658347 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43094: BlockingRecv error: recv got EOF from 127.0.0.1:43094 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 11:53:00.666909 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.664002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.1.210.61:43779)
0924 11:53:00.664166 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.664172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.665589 (+ 1417us) server_negotiation.cc:407] Connection header received
0924 11:53:00.665757 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.665763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.665843 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.665952 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.666667 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43108: BlockingRecv error: recv got EOF from 127.0.0.1:43108 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 11:53:00.703388 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.698913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.1.210.61:43779)
0924 11:53:00.699101 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.699107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.701805 (+ 2698us) server_negotiation.cc:407] Connection header received
0924 11:53:00.702018 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.702024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.702119 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.702235 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.703147 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43120: BlockingRecv error: recv got EOF from 127.0.0.1:43120 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 11:53:00.714610 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.711801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.1.210.61:43779)
0924 11:53:00.712028 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.712037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.712431 (+ 394us) server_negotiation.cc:407] Connection header received
0924 11:53:00.712645 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.712651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.712744 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.712853 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.714349 (+ 1496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43136: BlockingRecv error: recv got EOF from 127.0.0.1:43136 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 11:53:00.727635 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.725484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.1.210.61:43779)
0924 11:53:00.725665 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.725672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.726182 (+ 510us) server_negotiation.cc:407] Connection header received
0924 11:53:00.726366 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.726372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.726484 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.726609 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.727395 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43146: BlockingRecv error: recv got EOF from 127.0.0.1:43146 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:53:00.750011 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.747463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.1.210.61:43779)
0924 11:53:00.747656 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.747662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.748181 (+ 519us) server_negotiation.cc:407] Connection header received
0924 11:53:00.748468 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.748474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.748613 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.748728 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.749769 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43150: BlockingRecv error: recv got EOF from 127.0.0.1:43150 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 11:53:00.788362 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.786051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.1.210.61:43779)
0924 11:53:00.786246 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.786253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.786788 (+ 535us) server_negotiation.cc:407] Connection header received
0924 11:53:00.787000 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.787006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.787105 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.787234 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.788107 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43152: BlockingRecv error: recv got EOF from 127.0.0.1:43152 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 11:53:00.858744 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.856459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.1.210.61:43779)
0924 11:53:00.856703 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.856712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.857298 (+ 586us) server_negotiation.cc:407] Connection header received
0924 11:53:00.857535 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.857539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.857619 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.857733 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.858485 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43162: BlockingRecv error: recv got EOF from 127.0.0.1:43162 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:53:00.949891 24474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9632901ad0d644549605ffa7f719fc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9632901ad0d644549605ffa7f719fc67" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 46301 } } peers { permanent_uuid: "438f414d5e1d4989a7801a6bed270ab7" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 43779 } } peers { permanent_uuid: "9d9df49427bf49da8364c83289c207de" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 45787 } } }
I20260924 11:53:00.950174 24474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9632901ad0d644549605ffa7f719fc67 [sys.catalog]: This master's current role is: LEADER
W20260924 11:53:00.996140 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:00.994471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.1.210.61:43779)
0924 11:53:00.994662 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:00.994667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:00.994946 (+ 279us) server_negotiation.cc:407] Connection header received
0924 11:53:00.995106 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:00.995110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:00.995191 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:00.995289 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:00.995973 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43176: BlockingRecv error: recv got EOF from 127.0.0.1:43176 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 11:53:01.257686 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:01.255855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.1.210.61:43779)
0924 11:53:01.256092 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:01.256099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:01.256262 (+ 163us) server_negotiation.cc:407] Connection header received
0924 11:53:01.256504 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:01.256508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:01.256587 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:01.256701 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:01.257492 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43180: BlockingRecv error: recv got EOF from 127.0.0.1:43180 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:53:01.288297 24458 catalog_manager.cc:1259] Loaded cluster ID: 5783a281de454a99be82557e86a22c00
I20260924 11:53:01.288399 24458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: loading cluster ID for follower catalog manager: success
I20260924 11:53:01.290839 24458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: acquiring CA information for follower catalog manager: success
I20260924 11:53:01.292465 24458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d9df49427bf49da8364c83289c207de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:53:01.611236 24508 catalog_manager.cc:1259] Loaded cluster ID: 5783a281de454a99be82557e86a22c00
I20260924 11:53:01.611356 24508 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: loading cluster ID for follower catalog manager: success
I20260924 11:53:01.614014 24508 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: acquiring CA information for follower catalog manager: success
I20260924 11:53:01.615841 24508 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 438f414d5e1d4989a7801a6bed270ab7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:53:01.786003 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:01.778862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.1.210.61:43779)
0924 11:53:01.779681 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:01.779685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:01.779704 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:53:01.779768 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:01.779772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:01.779911 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:01.780096 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:01.780671 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.781914 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.783283 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.783984 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.784119 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:01.784609 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:01.785347 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:01.785487 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:01.785555 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 11:53:01.785643 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":559,"threads_started":1}
I20260924 11:53:01.798676 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:01.792396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.1.210.60:45787)
0924 11:53:01.792976 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:01.792981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:01.792999 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:53:01.793141 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:01.793145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:01.793293 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:01.793426 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:01.793896 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.794594 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.796528 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.796909 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.796996 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:01.797526 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:01.797996 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:01.798111 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:01.798259 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 11:53:01.798319 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":375,"threads_started":1}
I20260924 11:53:01.828132 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:01.806670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.1.210.62:46301)
0924 11:53:01.819086 (+ 12416us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:01.819094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:01.819865 (+ 771us) server_negotiation.cc:407] Connection header received
0924 11:53:01.820198 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:01.820206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:01.820455 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:01.820664 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:01.821292 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.822459 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.824774 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.825459 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.825596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:01.826404 (+ 808us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:01.826908 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:01.827022 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:01.827708 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 11:53:01.827795 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12264,"thread_start_us":12474,"threads_started":1}
I20260924 11:53:01.830578 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:01.806850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.1.210.61:43779)
0924 11:53:01.807040 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:01.807047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:01.821745 (+ 14698us) server_negotiation.cc:407] Connection header received
0924 11:53:01.822019 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:01.822025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:01.822128 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:01.822196 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:01.823257 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.824787 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.827975 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.828815 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.828927 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:01.829699 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:01.829830 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:01.829899 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:01.830117 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 11:53:01.830195 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:53:01.832999 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24223
I20260924 11:53:01.834097 24285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:53:01.837247 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:01.821983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.1.210.60:45787)
0924 11:53:01.822150 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:01.822158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:01.827622 (+ 5464us) server_negotiation.cc:407] Connection header received
0924 11:53:01.829402 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:01.829410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:01.829515 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:01.829593 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:01.831400 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.832413 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.834413 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:01.835259 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:01.835394 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:01.836325 (+ 931us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:01.836491 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:01.836594 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:01.836843 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 11:53:01.836942 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:53:01.952092 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24223
I20260924 11:53:01.981659 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24297
I20260924 11:53:01.982824 24359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:53:02.096133 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24297
I20260924 11:53:02.112428 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24371
I20260924 11:53:02.113608 24433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:53:02.219965 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24371
2026-09-24T11:53:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4827 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:53:02 dist-test-slave-dd32 krb5kdc[24594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:53:02 dist-test-slave-dd32 krb5kdc[24594](info): set up 2 sockets
Sep 24 11:53:02 dist-test-slave-dd32 krb5kdc[24594](info): commencing operation
krb5kdc: starting...
W20260924 11:53:03.325713 1864 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/test-user.keytab.
Sep 24 11:53:03 dist-test-slave-dd32 krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
Entry for principal kudu/127.1.210.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab.
I20260924 11:53:03.585950 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39719
--webserver_interface=127.1.210.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:39719,127.1.210.61:39959,127.1.210.60:40331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:53:03.911474 24607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:53:03.911873 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:53:03.912015 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:53:03.912160 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:53:03.919220 24607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:53:03.919343 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:53:03.919451 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:53:03.919548 24607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:53:03.919636 24607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:53:03.929435 24607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:39719,127.1.210.61:39959,127.1.210.60:40331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.62.keytab
--principal=kudu/127.1.210.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.62:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:53:03.930748 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:53:03.932055 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:53:03.943912 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:03.943873 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:03.943939 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:53:03.944684 24607 server_base.cc:1034] running on GCE node
I20260924 11:53:03.945528 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:53:03.946944 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:53:03.948215 24607 hybrid_clock.cc:648] HybridClock initialized: now 1790250783948164 us; error 56 us; skew 500 ppm
I20260924 11:53:03.948567 24607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:53:03 dist-test-slave-dd32 krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:53:03.951833 24607 init.cc:377] Logged in from keytab as kudu/127.1.210.62@KRBTEST.COM (short username kudu)
I20260924 11:53:03.953428 24607 webserver.cc:460] Webserver started at http://127.1.210.62:44981/ using document root <none> and password file <none>
I20260924 11:53:03.954108 24607 fs_manager.cc:359] Metadata directory not provided
I20260924 11:53:03.954236 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:53:03.954553 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:53:03.957640 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data/instance:
uuid: "a378c49910774f6a9bc2e2f4c74d7cfc"
format_stamp: "Formatted at 2026-09-24 11:53:03 on dist-test-slave-dd32"
server_key: "2629085f6171e9d2742cc32bd2452d83"
server_key_iv: "5051fcfa1cdd2fd95f68f162ef2cfa2d"
server_key_version: "encryptionkey@0"
I20260924 11:53:03.958410 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal/instance:
uuid: "a378c49910774f6a9bc2e2f4c74d7cfc"
format_stamp: "Formatted at 2026-09-24 11:53:03 on dist-test-slave-dd32"
server_key: "2629085f6171e9d2742cc32bd2452d83"
server_key_iv: "5051fcfa1cdd2fd95f68f162ef2cfa2d"
server_key_version: "encryptionkey@0"
I20260924 11:53:03.963415 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:53:03.966692 24622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:53:03.967904 24607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 11:53:03.968088 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal
uuid: "a378c49910774f6a9bc2e2f4c74d7cfc"
format_stamp: "Formatted at 2026-09-24 11:53:03 on dist-test-slave-dd32"
server_key: "2629085f6171e9d2742cc32bd2452d83"
server_key_iv: "5051fcfa1cdd2fd95f68f162ef2cfa2d"
server_key_version: "encryptionkey@0"
I20260924 11:53:03.968302 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:53:03.980492 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:53:03.984134 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:53:03.984395 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:53:04.002424 24607 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.62:39719
I20260924 11:53:04.002447 24673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.62:39719 every 8 connection(s)
I20260924 11:53:04.003750 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/data/info.pb
I20260924 11:53:04.007372 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:53:04.010215 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24607
I20260924 11:53:04.010433 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-0/wal/instance
I20260924 11:53:04.010865 1864 external_mini_cluster.cc:1428] Setting key 0c0322754b5bc3f85e06e901f86f07a9
I20260924 11:53:04.013988 24674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:04 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.61@KRBTEST.COM" created.
I20260924 11:53:04.039335 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.017174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39719 (local address 127.0.0.1:45056)
0924 11:53:04.018269 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.018302 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:04.020873 (+ 2571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:04.021832 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:04.021842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:04.022645 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:04.023120 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.023146 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.024677 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.024686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.026604 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.026622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.027545 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.027554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.027778 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.028480 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:04.028521 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:04.031633 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:04.035143 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.035158 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.035173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.035797 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.036208 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.036214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.036218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.036446 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.037318 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:04.037324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:04.037527 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 11:53:04.037747 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 11:53:04.038199 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":681,"threads_started":1}
I20260924 11:53:04.039340 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.017173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.1.210.62:39719)
0924 11:53:04.017990 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:04.018010 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:04.018435 (+ 425us) server_negotiation.cc:407] Connection header received
0924 11:53:04.021344 (+ 2909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:04.021349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:04.021582 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:04.021747 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:04.023331 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.024495 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.026739 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.027280 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.027405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.031883 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:04.031892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:04.031906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:04.031940 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:04.034995 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.035930 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.035936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.035939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.036079 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.036563 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.036566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.036568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.037188 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:04.037353 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:04.037941 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 11:53:04.038311 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":561,"thread_start_us":349,"threads_started":1}
I20260924 11:53:04.042273 24674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:53:04.043424 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.043029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49302)
0924 11:53:04.043209 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.043309 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 11:53:04.043812 24625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
I20260924 11:53:04.045362 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.044986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49314)
0924 11:53:04.045192 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.045254 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:53:04.045670 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:39959: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab.
Entry for principal kudu/127.1.210.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab.
I20260924 11:53:04.070015 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:39959
--webserver_interface=127.1.210.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab
--principal=kudu/127.1.210.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:39719,127.1.210.61:39959,127.1.210.60:40331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:53:04.095124 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } attempt: 1
I20260924 11:53:04.096321 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.095876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49320)
0924 11:53:04.096073 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.096171 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 11:53:04.097366 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.097002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49332)
0924 11:53:04.097180 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.097245 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:53:04.097584 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:39959: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
I20260924 11:53:04.165856 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } attempt: 2
I20260924 11:53:04.167119 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.166643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49342)
0924 11:53:04.166876 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.166972 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 11:53:04.168097 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.167765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49348)
0924 11:53:04.167889 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.167952 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:53:04.168308 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:39959: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
I20260924 11:53:04.259549 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } attempt: 3
I20260924 11:53:04.260804 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.260378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49356)
0924 11:53:04.260579 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.260668 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:53:04.261845 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.261525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49362)
0924 11:53:04.261660 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.261726 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:53:04.262080 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:39959: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
W20260924 11:53:04.394795 24681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:53:04.395097 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:53:04.395159 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:53:04.395241 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:53:04.402380 24681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:53:04.402467 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:53:04.402522 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:53:04.402571 24681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:53:04.402644 24681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:53:04.405326 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } attempt: 4
I20260924 11:53:04.406540 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.406015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49368)
0924 11:53:04.406314 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.406417 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260924 11:53:04.407553 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.407241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49376)
0924 11:53:04.407368 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.407433 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:53:04.407780 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.61:39959: Network error: Client connection negotiation failed: client connection to 127.1.210.61:39959: connect: Connection refused (error 111)
I20260924 11:53:04.413221 24681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:39719,127.1.210.61:39959,127.1.210.60:40331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.61.keytab
--principal=kudu/127.1.210.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.61:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:53:04.414469 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:53:04.415700 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:53:04.427318 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:04.427353 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:04.427278 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:53:04.428193 24681 server_base.cc:1034] running on GCE node
I20260924 11:53:04.428871 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:53:04.430295 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:53:04.431504 24681 hybrid_clock.cc:648] HybridClock initialized: now 1790250784431432 us; error 55 us; skew 500 ppm
I20260924 11:53:04.431780 24681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:53:04 dist-test-slave-dd32 krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:53:04.434885 24681 init.cc:377] Logged in from keytab as kudu/127.1.210.61@KRBTEST.COM (short username kudu)
I20260924 11:53:04.436335 24681 webserver.cc:460] Webserver started at http://127.1.210.61:39047/ using document root <none> and password file <none>
I20260924 11:53:04.436995 24681 fs_manager.cc:359] Metadata directory not provided
I20260924 11:53:04.437146 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:53:04.437461 24681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:53:04.440294 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data/instance:
uuid: "feee685ba01b40db8edda816946d2589"
format_stamp: "Formatted at 2026-09-24 11:53:04 on dist-test-slave-dd32"
server_key: "e1a8ef342b8fa21bbfc3eb2cd6f9d482"
server_key_iv: "7992f20068d0e4ffea8fa548f3936d82"
server_key_version: "encryptionkey@0"
I20260924 11:53:04.441039 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal/instance:
uuid: "feee685ba01b40db8edda816946d2589"
format_stamp: "Formatted at 2026-09-24 11:53:04 on dist-test-slave-dd32"
server_key: "e1a8ef342b8fa21bbfc3eb2cd6f9d482"
server_key_iv: "7992f20068d0e4ffea8fa548f3936d82"
server_key_version: "encryptionkey@0"
I20260924 11:53:04.446169 24681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:53:04.449468 24696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:53:04.450718 24681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:53:04.450892 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal
uuid: "feee685ba01b40db8edda816946d2589"
format_stamp: "Formatted at 2026-09-24 11:53:04 on dist-test-slave-dd32"
server_key: "e1a8ef342b8fa21bbfc3eb2cd6f9d482"
server_key_iv: "7992f20068d0e4ffea8fa548f3936d82"
server_key_version: "encryptionkey@0"
I20260924 11:53:04.451109 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:53:04.474647 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:53:04.478665 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:53:04.478894 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:53:04.496327 24681 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.61:39959
I20260924 11:53:04.496354 24747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.61:39959 every 8 connection(s)
I20260924 11:53:04.497753 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/data/info.pb
I20260924 11:53:04.501405 24748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:53:04.504134 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24681
I20260924 11:53:04.504312 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-1/wal/instance
I20260924 11:53:04.504776 1864 external_mini_cluster.cc:1428] Setting key cb82c51e01a5883195e9c106fcd3fea8
I20260924 11:53:04.507859 24748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:04 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.60@KRBTEST.COM; defaulting to no policy
I20260924 11:53:04.530179 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.510441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.1.210.62:39719)
0924 11:53:04.510688 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:04.510696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:04.511924 (+ 1228us) server_negotiation.cc:407] Connection header received
0924 11:53:04.513989 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:04.513998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:04.514106 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:04.514219 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:04.515619 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.516737 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.518263 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.519014 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.519163 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.522666 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:04.522675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:04.522683 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:04.522727 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:04.526902 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.527704 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.527710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.527714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.527868 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.528277 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.528283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.528288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.528673 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:04.528843 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:04.529783 (+ 940us) server_negotiation.cc:299] Negotiation successful
0924 11:53:04.529949 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:53:04.531167 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.510902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39719 (local address 127.0.0.1:45062)
0924 11:53:04.511742 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.511790 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:04.513712 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:04.514404 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:04.514411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:04.514923 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:04.515446 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.515462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.516881 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.516885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.518123 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.518135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.519204 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.519209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.519397 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.520079 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:04.520102 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:04.522444 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:04.527084 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.527091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.527104 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.527560 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.527990 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.527993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.527996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.528152 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.528898 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:04.528904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:04.529320 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 11:53:04.529620 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 11:53:04.530007 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":317,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.60@KRBTEST.COM" created.
I20260924 11:53:04.532856 24748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:04 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:53:04.549011 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.533694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49380)
0924 11:53:04.533910 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.533929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:04.534035 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:04.535081 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:04.535084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:04.535424 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:04.535554 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.535562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.537503 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.537506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.538336 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.538346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.539527 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.539531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.539647 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.540248 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:04.540264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:04.542524 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:04.546745 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.546749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.546752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.547095 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.547497 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.547500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.547502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.547644 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.548365 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:04.548368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:04.548466 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 11:53:04.548570 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 11:53:04.548763 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 11:53:04.549559 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.533901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.1.210.61:39959)
0924 11:53:04.534590 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:04.534610 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:04.534640 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:53:04.534695 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:04.534706 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:04.534931 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:04.535073 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:04.535723 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.537376 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.538479 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.539393 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.539534 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.542738 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:04.542753 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:04.542772 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:04.542817 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:04.546609 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.547219 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.547229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.547235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.547372 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.547748 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.547756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.547761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.548230 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:04.548443 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:04.548756 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 11:53:04.549025 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"spinlock_wait_cycles":22912,"thread_start_us":351,"threads_started":1}
I20260924 11:53:04.551783 24748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:53:04.552918 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.552505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60832)
0924 11:53:04.552685 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.552794 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:53:04.553207 24697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.554808 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.554474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60838)
0924 11:53:04.554626 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.554718 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:53:04.555059 24748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab.
Entry for principal kudu/127.1.210.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab.
I20260924 11:53:04.566272 1864 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1DchkY/build/asan/bin/kudu
/tmp/dist-test-task1DchkY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:40331
--webserver_interface=127.1.210.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab
--principal=kudu/127.1.210.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.62:39719,127.1.210.61:39959,127.1.210.60:40331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:53:04.604416 24748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 1
I20260924 11:53:04.605615 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.605164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60842)
0924 11:53:04.605355 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.605464 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:53:04.606561 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.606241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60854)
0924 11:53:04.606381 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.606465 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 11:53:04.606762 24748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.674988 24748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 2
I20260924 11:53:04.676082 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.675634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60864)
0924 11:53:04.675841 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.675934 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 11:53:04.677145 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.676747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60874)
0924 11:53:04.676898 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.676970 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:53:04.677374 24748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.707078 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } attempt: 5
Sep 24 11:53:04 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:53:04.722062 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.708008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.1.210.61:39959)
0924 11:53:04.708185 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:04.708190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:04.708209 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:53:04.708413 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:04.708416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:04.708477 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:04.708559 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:04.709571 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.710289 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.711992 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.712404 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.712494 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.716570 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:04.716576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:04.716579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:04.716618 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:04.719235 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.720140 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.720143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.720146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.720256 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:04.720701 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:04.720704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:04.720706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:04.720974 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:04.721157 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:04.721664 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 11:53:04.721845 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:53:04.722162 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.707883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49388)
0924 11:53:04.708119 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.708141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:04.708269 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:04.708630 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:04.708635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:04.709182 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:04.709425 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.709436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.710439 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.710445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.711843 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:04.711861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:04.712576 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:04.712583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:04.712805 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:04.713489 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:04.713529 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:04.716365 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:04.719379 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.719386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.719392 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.720000 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.720378 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:04.720384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:04.720389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:04.720585 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:04.721139 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:04.721146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:04.721311 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 11:53:04.721507 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 11:53:04.721796 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:53:04.723307 24674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:53:04.724380 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.724012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60880)
0924 11:53:04.724178 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.724247 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 11:53:04.725395 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.725055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60882)
0924 11:53:04.725222 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.725287 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:53:04.725662 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.768612 24748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 3
I20260924 11:53:04.769801 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.769332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60892)
0924 11:53:04.769555 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.769643 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:53:04.770829 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.770422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60906)
0924 11:53:04.770640 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.770717 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 11:53:04.771023 24748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.776892 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 1
I20260924 11:53:04.777933 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.777531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60914)
0924 11:53:04.777707 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.777791 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:53:04.778846 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.778529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60928)
0924 11:53:04.778684 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.778746 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:53:04.779059 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.847273 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 2
I20260924 11:53:04.848729 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.848159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60944)
0924 11:53:04.848411 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.848558 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 11:53:04.849795 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.849452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60960)
0924 11:53:04.849620 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.849691 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 11:53:04.850014 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
W20260924 11:53:04.901222 24755 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:53:04.901645 24755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:53:04.901786 24755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:53:04.901945 24755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:53:04.909608 24755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:53:04.909740 24755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:53:04.909848 24755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:53:04.909955 24755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:53:04.910054 24755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:53:04.914233 24748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 4
I20260924 11:53:04.915313 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.914815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60962)
0924 11:53:04.915089 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.915169 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 11:53:04.916216 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.915916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60966)
0924 11:53:04.916057 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.916123 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:53:04.916404 24748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.920362 24755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.62:39719,127.1.210.61:39959,127.1.210.60:40331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/krb5kdc/kudu_127.1.210.60.keytab
--principal=kudu/127.1.210.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.60:40331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:53:04.921836 24755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:53:04.923225 24755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:53:04.937875 24763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:04.938081 24760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:53:04.938117 24761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:53:04.938866 24755 server_base.cc:1034] running on GCE node
I20260924 11:53:04.939636 24755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:53:04.940971 24755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:53:04.942224 24755 hybrid_clock.cc:648] HybridClock initialized: now 1790250784942149 us; error 51 us; skew 500 ppm
I20260924 11:53:04.942555 24755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:53:04 dist-test-slave-dd32 krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:53:04.945569 24755 init.cc:377] Logged in from keytab as kudu/127.1.210.60@KRBTEST.COM (short username kudu)
I20260924 11:53:04.946838 24755 webserver.cc:460] Webserver started at http://127.1.210.60:43779/ using document root <none> and password file <none>
I20260924 11:53:04.947455 24755 fs_manager.cc:359] Metadata directory not provided
I20260924 11:53:04.947535 24755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:53:04.947789 24755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:53:04.950757 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data/instance:
uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9"
format_stamp: "Formatted at 2026-09-24 11:53:04 on dist-test-slave-dd32"
server_key: "2f3cc3968ffe9d9560932d5a5cbf8fbf"
server_key_iv: "9c9ab9c8f73210370eaa8ef67960c387"
server_key_version: "encryptionkey@0"
I20260924 11:53:04.951468 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal/instance:
uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9"
format_stamp: "Formatted at 2026-09-24 11:53:04 on dist-test-slave-dd32"
server_key: "2f3cc3968ffe9d9560932d5a5cbf8fbf"
server_key_iv: "9c9ab9c8f73210370eaa8ef67960c387"
server_key_version: "encryptionkey@0"
I20260924 11:53:04.956228 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 3
I20260924 11:53:04.956554 24755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:53:04.957405 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.956876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60970)
0924 11:53:04.957166 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.957251 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 11:53:04.958560 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:04.958180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60972)
0924 11:53:04.958374 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:04.958452 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 11:53:04.958789 24674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.60:40331: Network error: Client connection negotiation failed: client connection to 127.1.210.60:40331: connect: Connection refused (error 111)
I20260924 11:53:04.960683 24770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:53:04.962064 24755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:53:04.962247 24755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data,/tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal
uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9"
format_stamp: "Formatted at 2026-09-24 11:53:04 on dist-test-slave-dd32"
server_key: "2f3cc3968ffe9d9560932d5a5cbf8fbf"
server_key_iv: "9c9ab9c8f73210370eaa8ef67960c387"
server_key_version: "encryptionkey@0"
I20260924 11:53:04.962466 24755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:53:04.984916 24755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:53:04.989277 24755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:53:04.989548 24755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:53:05.009208 24755 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.60:40331
I20260924 11:53:05.009227 24821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.60:40331 every 8 connection(s)
I20260924 11:53:05.010564 24755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/data/info.pb
I20260924 11:53:05.014067 24822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:53:05.016878 1864 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1DchkY/build/asan/bin/kudu as pid 24755
I20260924 11:53:05.017097 1864 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1DchkY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790250429340952-1864-0/minicluster-data/master-2/wal/instance
I20260924 11:53:05.017560 1864 external_mini_cluster.cc:1428] Setting key 0516e9bca5d4b7bf4ab907707695a595
I20260924 11:53:05.021780 24822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:53:05.039914 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.024934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.1.210.62:39719)
0924 11:53:05.025180 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.025186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.025928 (+ 742us) server_negotiation.cc:407] Connection header received
0924 11:53:05.026123 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.026129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.026201 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.026298 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.027750 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.028607 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.030454 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.030860 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.030953 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.034512 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.034517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.034520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.034549 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.037046 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.037708 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.037711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.037714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.037842 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.038360 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.038364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.038366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.038613 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.038731 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.039354 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.039578 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:53:05.041078 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.024904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.62:39719 (local address 127.0.0.1:45072)
0924 11:53:05.025780 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:05.025825 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:05.025979 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:05.026355 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:05.026360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:05.027211 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:05.027587 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.027604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.028791 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.028795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.030310 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.030326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.031091 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.031100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.031304 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.031922 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:05.031952 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:05.034320 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:05.037183 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.037190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.037198 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.037554 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.037980 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.037983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.037985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.038126 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.038735 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:05.038742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:05.038973 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 11:53:05.039232 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:53:05.039967 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":285,"thread_start_us":389,"threads_started":1}
I20260924 11:53:05.041116 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.019408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.1.210.60:40331)
0924 11:53:05.020154 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.020169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.020215 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:53:05.022310 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.022339 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.022864 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.023267 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.024749 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.026512 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.028231 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.029150 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.029281 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.033906 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.033916 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.033926 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.033973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.036959 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.037664 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.037675 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.037678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.037828 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.038292 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.038299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.038303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.038826 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.039130 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.039294 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.039737 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":521,"thread_start_us":327,"threads_started":1}
I20260924 11:53:05.043375 24822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.62@KRBTEST.COM
I20260924 11:53:05.057006 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.044171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.61:39959 (local address 127.0.0.1:49402)
0924 11:53:05.044318 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:05.044336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:05.044422 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:05.044777 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:05.044781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:05.045227 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:05.045359 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.045367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.046328 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.046333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.047191 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.047201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.047896 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.047902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.048110 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.048719 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:05.048744 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:05.051033 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:05.054533 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.054538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.054542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.054890 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.055325 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.055331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.055334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.055537 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.056062 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:05.056067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:05.056221 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 11:53:05.056357 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 11:53:05.056636 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 11:53:05.057046 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.044211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.1.210.61:39959)
0924 11:53:05.044405 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.044411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.044429 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:53:05.044562 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.044568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.044642 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.044762 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.045503 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.046172 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.047333 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.047750 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.047877 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.051224 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.051231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.051236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.051278 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.054400 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.055013 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.055019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.055023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.055206 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.055651 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.055658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.055661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.055929 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.056098 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.056461 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.056658 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:53:05.058154 24822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:53:05.058398 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.044009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.1.210.62:39719)
0924 11:53:05.044544 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.044547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.044559 (+ 12us) server_negotiation.cc:407] Connection header received
0924 11:53:05.044600 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.044603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.044650 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.044772 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.045897 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.046631 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.047795 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.048181 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.048313 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.052583 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.052588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.052592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.052623 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.055269 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.055891 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.055898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.055901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.056030 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.056391 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.056397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.056400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.056664 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.056945 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.057855 (+ 910us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.058076 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.60@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.61@KRBTEST.COM
I20260924 11:53:05.072113 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.058973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.1.210.60:40331)
0924 11:53:05.059154 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.059158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.059307 (+ 149us) server_negotiation.cc:407] Connection header received
0924 11:53:05.059467 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.059472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.059578 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.059663 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.060527 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.061343 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.062618 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.063069 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.063158 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.066438 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.066443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.066445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.066471 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.069514 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.070285 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.070291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.070294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.070423 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.070782 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.070789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.070792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.071038 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.071187 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.071512 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.071744 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260924 11:53:05.072292 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.058981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:60994)
0924 11:53:05.059223 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:05.059236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:05.059344 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:05.059719 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:05.059723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:05.060204 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:05.060386 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.060399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.061481 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.061488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.062485 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.062496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.063211 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.063217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.063433 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.063951 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:05.063974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:05.066258 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:05.069627 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.069634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.069638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.070159 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.070528 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.070534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.070538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.070680 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.071152 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:05.071157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:05.071292 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 11:53:05.071451 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 11:53:05.071888 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":152}
I20260924 11:53:05.074738 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.060171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.1.210.61:39959)
0924 11:53:05.060312 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.060315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.060329 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:53:05.060440 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.060443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.060527 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.060628 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.061893 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.062605 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.064105 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.064503 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.064591 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.069319 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.069323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.069326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.069355 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.072041 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.072759 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.072764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.072767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.072938 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.073343 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.073349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.073352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.073665 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.073928 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.074199 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.074430 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:53:05.077870 24822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: Bootstrap starting.
I20260924 11:53:05.079315 1864 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:53:05.081508 24822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: Neither blocks nor log segments found. Creating new log.
I20260924 11:53:05.082834 24822 log.cc:824] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:53:05.081590 24649 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45082
I20260924 11:53:05.086166 24822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: No bootstrap required, opened a new log
I20260924 11:53:05.096874 24822 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.097357 24822 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c5e79b382d4e8aa1315ab0ede8d3f9, State: Initialized, Role: FOLLOWER
I20260924 11:53:05.097849 24822 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.098809 24828 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.099201 24828 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.100041 24822 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:53:05.109509 24839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:53:05.109647 24839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:53:05.107509 24755 master_runner.cc:186] Error getting master registration for 127.1.210.62:39719: OK, instance_id { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" instance_seqno: 1790250783997232 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:53:05.118615 24755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:53:05.135998 24674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 4
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250783, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.62@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:53:05.152184 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.136806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.1.210.60:40331)
0924 11:53:05.137043 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.137049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.137115 (+ 66us) server_negotiation.cc:407] Connection header received
0924 11:53:05.137312 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.137318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.137401 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.137513 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.138508 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.139658 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.141585 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.142367 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.142491 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.146459 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.146464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.146467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.146503 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.149394 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.150058 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.150069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.150071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.150195 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.150589 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.150603 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.150605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.150960 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.151129 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.151565 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.151801 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 11:53:05.152182 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.136737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:32768)
0924 11:53:05.136976 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:05.136999 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:05.137166 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:05.137577 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:05.137583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:05.138132 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:05.138353 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.138367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.139843 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.139849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.141427 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.141444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.142537 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.142544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.142752 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.143459 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:05.143484 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:05.146264 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:05.149530 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.149535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.149540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.149929 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.150310 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.150315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.150319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.150469 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.151114 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:05.151119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:05.151275 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:53:05.151445 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 11:53:05.151780 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 11:53:05.157450 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: Bootstrap starting.
I20260924 11:53:05.160786 24674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: Neither blocks nor log segments found. Creating new log.
I20260924 11:53:05.161897 24674 log.cc:824] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: Log is configured to *not* fsync() on all Append() calls
I20260924 11:53:05.164621 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: No bootstrap required, opened a new log
I20260924 11:53:05.170192 24674 raft_consensus.cc:348] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.170532 24674 raft_consensus.cc:729] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378c49910774f6a9bc2e2f4c74d7cfc, State: Initialized, Role: FOLLOWER
I20260924 11:53:05.171020 24674 consensus_queue.cc:260] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.172070 24842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.172497 24842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.173306 24674 sys_catalog.cc:564] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:53:05.173487 24649 raft_consensus.cc:482] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 11:53:05.173748 24649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:53:05.177680 24649 raft_consensus.cc:504] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.181522 24649 leader_election.cc:288] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [CANDIDATE]: Term 1 election: Requested vote from peers feee685ba01b40db8edda816946d2589 (127.1.210.61:39959), f7c5e79b382d4e8aa1315ab0ede8d3f9 (127.1.210.60:40331)
I20260924 11:53:05.181789 24797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9"
I20260924 11:53:05.182417 24797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:53:05.187197 24797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a378c49910774f6a9bc2e2f4c74d7cfc in term 1.
I20260924 11:53:05.187988 24623 leader_election.cc:302] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a378c49910774f6a9bc2e2f4c74d7cfc, f7c5e79b382d4e8aa1315ab0ede8d3f9; no voters:
I20260924 11:53:05.188575 24842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:53:05.213527 24842 raft_consensus.cc:686] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [term 1 LEADER]: Becoming Leader. State: Replica: a378c49910774f6a9bc2e2f4c74d7cfc, State: Running, Role: LEADER
I20260924 11:53:05.214378 24842 consensus_queue.cc:237] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.215587 24748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } attempt: 5
Sep 24 11:53:05 dist-test-slave-dd32 krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790250784, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.61@KRBTEST.COM for kudu/127.1.210.60@KRBTEST.COM
I20260924 11:53:05.236940 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.216303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.60:40331 (local address 127.0.0.1:32772)
0924 11:53:05.216738 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 11:53:05.216778 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:53:05.216926 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:53:05.217966 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:53:05.217971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:53:05.218582 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:53:05.218855 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.218873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.220448 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.220454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.221909 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:53:05.221926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.223593 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.223601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:53:05.223868 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.224625 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:53:05.224662 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:53:05.228544 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:53:05.233347 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.233355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.233361 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.234046 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.234577 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:53:05.234586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:53:05.234592 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:53:05.234793 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:53:05.235607 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:53:05.235615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:53:05.235823 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 11:53:05.236008 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 11:53:05.236376 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
W20260924 11:53:05.237233 24628 tablet.cc:2369] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:53:05.238131 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.217189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.1.210.60:40331)
0924 11:53:05.217386 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.217396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.217444 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:53:05.217513 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.217522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.217627 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.217741 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.219069 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.220226 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.222096 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.222856 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.223002 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.228812 (+ 5810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.228820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.228825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.228875 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.233164 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.234212 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.234220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.234227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.234409 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.234941 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.234948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.234952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.235354 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.235535 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.237347 (+ 1812us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.237664 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 11:53:05.241536 24776 tablet.cc:2369] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:53:05.191009 24607 master_runner.cc:186] Error getting master registration for 127.1.210.61:39959: OK, instance_id { permanent_uuid: "feee685ba01b40db8edda816946d2589" instance_seqno: 1790250784491443 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:53:05.248628 24607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 11:53:05.268502 24853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:53:05.268698 24853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:53:05.270318 24748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: Bootstrap starting.
I20260924 11:53:05.274354 24856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a378c49910774f6a9bc2e2f4c74d7cfc. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.274829 24856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: This master's current role is: LEADER
I20260924 11:53:05.275018 24748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: Neither blocks nor log segments found. Creating new log.
I20260924 11:53:05.276542 24748 log.cc:824] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: Log is configured to *not* fsync() on all Append() calls
I20260924 11:53:05.290263 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.224331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.1.210.60:40331)
0924 11:53:05.238393 (+ 14062us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.238398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.259822 (+ 21424us) server_negotiation.cc:407] Connection header received
0924 11:53:05.260242 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.260252 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.260358 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.260434 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.269245 (+ 8811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.270697 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.276164 (+ 5467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.277005 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.277178 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.281594 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.281602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.281607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.281661 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.286563 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.287498 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.287506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.287513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.287665 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.288179 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.288187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.288190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.288604 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.288901 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.289558 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.289825 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13935,"thread_start_us":33410,"threads_started":1}
I20260924 11:53:05.292253 24865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:53:05.294813 24865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:53:05.302601 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.222592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.1.210.61:39959)
0924 11:53:05.222757 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.222764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.254181 (+ 31417us) server_negotiation.cc:407] Connection header received
0924 11:53:05.254745 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.254752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.254854 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.254932 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.277750 (+ 22818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.278838 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.282968 (+ 4130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.283655 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.283751 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.294309 (+ 10558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.294316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.294323 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.294362 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.297966 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.298832 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.298838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.298841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.298987 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.299451 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.299457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.299461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.299830 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.300050 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.301925 (+ 1875us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.302202 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:53:05.303849 24748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: No bootstrap required, opened a new log
I20260924 11:53:05.305563 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.245497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.1.210.62:39719)
0924 11:53:05.245689 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.245696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.264567 (+ 18871us) server_negotiation.cc:407] Connection header received
0924 11:53:05.265692 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.265702 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.265791 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.265870 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.276316 (+ 10446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.277414 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.284759 (+ 7345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.285479 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.285586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.294303 (+ 8717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:53:05.294311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:53:05.294318 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:53:05.294388 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:53:05.299476 (+ 5088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.300877 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.300884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.300919 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.301325 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:53:05.302218 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:53:05.302251 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:53:05.302256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:53:05.303314 (+ 1058us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:53:05.303737 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.304826 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.305135 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:53:05.316545 24748 raft_consensus.cc:348] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.316962 24748 raft_consensus.cc:729] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feee685ba01b40db8edda816946d2589, State: Initialized, Role: FOLLOWER
I20260924 11:53:05.317497 24748 consensus_queue.cc:260] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } }
I20260924 11:53:05.318488 24868 sys_catalog.cc:455] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.318882 24868 sys_catalog.cc:458] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.320660 24748 sys_catalog.cc:564] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:53:05.320818 24723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "feee685ba01b40db8edda816946d2589"
I20260924 11:53:05.321337 24723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:53:05.328676 24723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a378c49910774f6a9bc2e2f4c74d7cfc in term 1.
I20260924 11:53:05.333438 24723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [term 1 FOLLOWER]: Refusing update from remote peer a378c49910774f6a9bc2e2f4c74d7cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:53:05.333740 24797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [term 1 FOLLOWER]: Refusing update from remote peer a378c49910774f6a9bc2e2f4c74d7cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:53:05.334705 24842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:53:05.335645 24842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:53:05.353590 24828 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a378c49910774f6a9bc2e2f4c74d7cfc. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.353940 24828 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.361776 24842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.362197 24842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: This master's current role is: LEADER
I20260924 11:53:05.365635 24872 mvcc.cc:205] Tried to move back new op lower bound from 7332867216715317248 to 7332867216245456896. Current Snapshot: MvccSnapshot[applied={T|T < 7332867216715317248}]
I20260924 11:53:05.366218 24865 catalog_manager.cc:1347] Generated new cluster ID: 2fad2e9f4a8440df9a73bb1eca815643
I20260924 11:53:05.366328 24865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:53:05.377725 24828 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.378016 24828 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.394474 24868 sys_catalog.cc:455] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a378c49910774f6a9bc2e2f4c74d7cfc. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.394814 24868 sys_catalog.cc:458] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.395916 24856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.396171 24856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc [sys.catalog]: This master's current role is: LEADER
I20260924 11:53:05.398736 24868 sys_catalog.cc:455] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378c49910774f6a9bc2e2f4c74d7cfc" member_type: VOTER last_known_addr { host: "127.1.210.62" port: 39719 } } peers { permanent_uuid: "feee685ba01b40db8edda816946d2589" member_type: VOTER last_known_addr { host: "127.1.210.61" port: 39959 } } peers { permanent_uuid: "f7c5e79b382d4e8aa1315ab0ede8d3f9" member_type: VOTER last_known_addr { host: "127.1.210.60" port: 40331 } } }
I20260924 11:53:05.399034 24868 sys_catalog.cc:458] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:53:05.420251 24865 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:53:05.425817 24865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:53:05.441802 24865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a378c49910774f6a9bc2e2f4c74d7cfc: Generated new TSK 0
I20260924 11:53:05.442898 24865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:53:05.464042 24887 catalog_manager.cc:1259] Loaded cluster ID: 2fad2e9f4a8440df9a73bb1eca815643
I20260924 11:53:05.464155 24887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: loading cluster ID for follower catalog manager: success
I20260924 11:53:05.474062 24887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: acquiring CA information for follower catalog manager: success
I20260924 11:53:05.476986 24887 catalog_manager.cc:1602] T 00000000000000000000000000000000 P feee685ba01b40db8edda816946d2589: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:53:05.598806 24752 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.586367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.1.210.61:39959)
0924 11:53:05.586595 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.586601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.595590 (+ 8989us) server_negotiation.cc:407] Connection header received
0924 11:53:05.596830 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.596837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.596925 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.597002 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.598571 (+ 1569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49426: BlockingRecv error: recv got EOF from 127.0.0.1:49426 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 11:53:05.600292 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.588452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.1.210.60:40331)
0924 11:53:05.588637 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.588644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.596056 (+ 7412us) server_negotiation.cc:407] Connection header received
0924 11:53:05.597235 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.597241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.597326 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.597401 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.600056 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32792: BlockingRecv error: recv got EOF from 127.0.0.1:32792 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 11:53:05.602778 24676 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.590068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.1.210.62:39719)
0924 11:53:05.590242 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.590248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.600838 (+ 10590us) server_negotiation.cc:407] Connection header received
0924 11:53:05.601454 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.601459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.601564 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.601641 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.602530 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45090: BlockingRecv error: recv got EOF from 127.0.0.1:45090 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:53:05.653334 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.637608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.1.210.61:39959)
0924 11:53:05.637807 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.637813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.645238 (+ 7425us) server_negotiation.cc:407] Connection header received
0924 11:53:05.645787 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.645794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.646034 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.646184 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.646928 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.648101 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.650215 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.650902 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.651011 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.651784 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.652515 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.652638 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.652888 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.652986 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 11:53:05.700551 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.687535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.1.210.60:40331)
0924 11:53:05.687791 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.687801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.698405 (+ 10604us) server_negotiation.cc:407] Connection header received
0924 11:53:05.698972 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.698980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.699114 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.699189 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.700273 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32796: BlockingRecv error: recv got EOF from 127.0.0.1:32796 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:53:05.751578 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.736686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.1.210.61:39959)
0924 11:53:05.736926 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.736933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.744396 (+ 7463us) server_negotiation.cc:407] Connection header received
0924 11:53:05.745025 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.745033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.745152 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.745233 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.745976 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.747159 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.749291 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.749954 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.750078 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.750698 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.750858 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.750923 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.751158 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.751243 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 11:53:05.798882 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.785958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.1.210.60:40331)
0924 11:53:05.786168 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.786175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.796812 (+ 10637us) server_negotiation.cc:407] Connection header received
0924 11:53:05.797378 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.797385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.797494 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.797567 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.798640 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32802: BlockingRecv error: recv got EOF from 127.0.0.1:32802 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:53:05.850549 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.834941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.1.210.61:39959)
0924 11:53:05.835127 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.835133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.842471 (+ 7338us) server_negotiation.cc:407] Connection header received
0924 11:53:05.842958 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.842966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.844258 (+ 1292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.844361 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.845057 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.846274 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.848319 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.848990 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.849103 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.849690 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.849824 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.849887 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.850102 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.850200 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 11:53:05.867027 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.864649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.1.210.60:40331)
0924 11:53:05.864936 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.864944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.865512 (+ 568us) server_negotiation.cc:407] Connection header received
0924 11:53:05.865737 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.865743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.865856 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.865952 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.866791 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32810: BlockingRecv error: recv got EOF from 127.0.0.1:32810 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 11:53:05.879917 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.874203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.1.210.61:39959)
0924 11:53:05.874484 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.874492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.875003 (+ 511us) server_negotiation.cc:407] Connection header received
0924 11:53:05.875205 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.875213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.875306 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.875407 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.875844 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.876762 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.878205 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.878643 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.878784 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.879232 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.879336 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.879431 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.879544 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.879618 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260924 11:53:05.887385 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.885853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.1.210.60:40331)
0924 11:53:05.886093 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.886098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.886130 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:53:05.886202 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.886208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.886290 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.886393 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.887157 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32812: BlockingRecv error: recv got EOF from 127.0.0.1:32812 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260924 11:53:05.904724 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.898578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.1.210.61:39959)
0924 11:53:05.898828 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.898841 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.899378 (+ 537us) server_negotiation.cc:407] Connection header received
0924 11:53:05.899653 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.899658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.899738 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.899856 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.900304 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.901287 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.902786 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.903304 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.903445 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.903898 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.904044 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.904155 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.904291 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.904365 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 11:53:05.915153 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.910407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.1.210.60:40331)
0924 11:53:05.910629 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.910637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.913659 (+ 3022us) server_negotiation.cc:407] Connection header received
0924 11:53:05.913873 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.913880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.913970 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.914079 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.914889 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32822: BlockingRecv error: recv got EOF from 127.0.0.1:32822 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:53:05.940418 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.934160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.1.210.61:39959)
0924 11:53:05.934383 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.934393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.935130 (+ 737us) server_negotiation.cc:407] Connection header received
0924 11:53:05.935363 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.935368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.935434 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.935543 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.936235 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.937308 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.938603 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.939059 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.939213 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.939634 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.939737 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.939853 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.940016 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.940108 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 11:53:05.948256 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.946732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.1.210.60:40331)
0924 11:53:05.946912 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.946930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.946951 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:53:05.947034 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.947039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.947118 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.947226 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.948033 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32838: BlockingRecv error: recv got EOF from 127.0.0.1:32838 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:53:05.991523 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:05.985031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.1.210.61:39959)
0924 11:53:05.985301 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.985310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.985603 (+ 293us) server_negotiation.cc:407] Connection header received
0924 11:53:05.985771 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.985778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.985921 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.986055 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:05.986410 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.987621 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.988938 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:05.989754 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:05.989903 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:05.990263 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:05.990409 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:05.990536 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:05.990633 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 11:53:05.991128 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260924 11:53:05.999047 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:05.997301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.1.210.60:40331)
0924 11:53:05.997510 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:05.997517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:05.997807 (+ 290us) server_negotiation.cc:407] Connection header received
0924 11:53:05.997934 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:05.997941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:05.998037 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:05.998148 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:05.998813 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32842: BlockingRecv error: recv got EOF from 127.0.0.1:32842 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:53:06.071409 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:06.066268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.1.210.61:39959)
0924 11:53:06.066492 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.066499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.066775 (+ 276us) server_negotiation.cc:407] Connection header received
0924 11:53:06.067038 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.067044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.067128 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.067232 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:06.067659 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.068390 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.069698 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.070128 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.070263 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:06.070637 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:06.070744 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:06.070846 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:06.070994 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 11:53:06.071074 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 11:53:06.078651 24825 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:53:06.076880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.1.210.60:40331)
0924 11:53:06.077188 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.077194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.077471 (+ 277us) server_negotiation.cc:407] Connection header received
0924 11:53:06.077617 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.077622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.077690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.077771 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:53:06.078469 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32844: BlockingRecv error: recv got EOF from 127.0.0.1:32844 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:53:06.111513 24839 catalog_manager.cc:1259] Loaded cluster ID: 2fad2e9f4a8440df9a73bb1eca815643
I20260924 11:53:06.111622 24839 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: loading cluster ID for follower catalog manager: success
I20260924 11:53:06.114063 24839 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: acquiring CA information for follower catalog manager: success
I20260924 11:53:06.115736 24839 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7c5e79b382d4e8aa1315ab0ede8d3f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:53:06.214854 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:06.210227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.1.210.61:39959)
0924 11:53:06.210476 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.210482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.210727 (+ 245us) server_negotiation.cc:407] Connection header received
0924 11:53:06.210883 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.210888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.210956 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.211038 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:06.211426 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.212123 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.213367 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.213765 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.213853 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:06.214174 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:06.214284 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:06.214348 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:06.214471 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:53:06.214533 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:53:06.225553 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:06.220254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.1.210.60:40331)
0924 11:53:06.220505 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.220514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.220756 (+ 242us) server_negotiation.cc:407] Connection header received
0924 11:53:06.220897 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.220901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.221048 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.221207 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:06.221527 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.222232 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.223485 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.223878 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.223967 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:06.224435 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:06.224955 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:06.225124 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:06.225180 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 11:53:06.225242 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 11:53:06.240569 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:06.231135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.1.210.60:40331)
0924 11:53:06.231326 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.231335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.234547 (+ 3212us) server_negotiation.cc:407] Connection header received
0924 11:53:06.234759 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.234764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.234837 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.234890 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:06.235589 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.236562 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.238347 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.238997 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.239123 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:06.239836 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:06.239937 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:06.240043 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:06.240162 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 11:53:06.240238 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:53:06.240962 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:06.231166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.1.210.61:39959)
0924 11:53:06.231424 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.231433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.234092 (+ 2659us) server_negotiation.cc:407] Connection header received
0924 11:53:06.234428 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.234435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.234542 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.234638 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:06.235273 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.236285 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.238604 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.239046 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.239188 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:06.240205 (+ 1017us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:06.240337 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:06.240400 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:06.240641 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 11:53:06.240723 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:53:06.241631 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:53:06.230914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.1.210.62:39719)
0924 11:53:06.233524 (+ 2610us) server_negotiation.cc:206] Beginning negotiation
0924 11:53:06.233530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:53:06.233549 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:53:06.233872 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:53:06.233877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:53:06.234133 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:53:06.234302 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:53:06.235181 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.236022 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.238503 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:53:06.239159 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:53:06.239291 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:53:06.240269 (+ 978us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:53:06.240718 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:53:06.240837 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:53:06.241041 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 11:53:06.241415 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2491,"thread_start_us":2066,"threads_started":1}
I20260924 11:53:06.246801 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24607
I20260924 11:53:06.247915 24669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:53:06.357537 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24607
I20260924 11:53:06.379304 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24681
I20260924 11:53:06.380470 24743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:53:06.486876 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24681
I20260924 11:53:06.507704 1864 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1DchkY/build/asan/bin/kudu pid 24755
I20260924 11:53:06.508816 24817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:53:06.614526 1864 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1DchkY/build/asan/bin/kudu with pid 24755
2026-09-24T11:53:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4394 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9222 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357239 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST